2024-03-24T10:22:37.034 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T10:22:37.035 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049 --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/orig.config.yaml 2024-03-24T10:22:37.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:22:37.169 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049 branch: wip-batrick-testing-20240323.003144-squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} email: pdonnell@redhat.com first_in_suite: false job_id: '7619049' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 21b3bd672163092b4e542f772797fcba02d3d4de ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI9EvjocjrvBu7amPMhnD4sgSivbF05PbumcxFiTQTEAOnMr9s+stLL13DGeLbt4DBy4k42+xo17vHsmObc4vbo= smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCdFtkHL/LzKO/bqO1/8DiX80bAYuDCkHNfY0y+tyyywsuYpE5ok+2ZrGqubxi+op9W4OeJOGeX//+Qdh0erhwU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T10:22:37.169 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T10:22:37.170 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T10:22:37.170 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T10:22:37.172 INFO:teuthology.task.internal:Checking packages... 2024-03-24T10:22:37.192 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T10:22:37.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:22:37.193 INFO:teuthology.packaging:ref: None 2024-03-24T10:22:37.193 INFO:teuthology.packaging:tag: None 2024-03-24T10:22:37.193 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T10:22:37.193 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:22:37.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T10:22:37.406 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T10:22:37.408 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T10:22:37.415 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T10:22:37.415 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T10:22:37.421 INFO:teuthology.task.internal:Saving configuration 2024-03-24T10:22:37.437 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T10:22:37.443 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:22:37.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:17:57.552395', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI9EvjocjrvBu7amPMhnD4sgSivbF05PbumcxFiTQTEAOnMr9s+stLL13DGeLbt4DBy4k42+xo17vHsmObc4vbo='} 2024-03-24T10:22:37.493 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:17:57.554437', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCdFtkHL/LzKO/bqO1/8DiX80bAYuDCkHNfY0y+tyyywsuYpE5ok+2ZrGqubxi+op9W4OeJOGeX//+Qdh0erhwU='} 2024-03-24T10:22:37.493 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T10:22:37.499 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-24T10:22:37.499 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-24T10:22:37.500 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T10:22:37.572 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4fd6922160>, signals=[15]) 2024-03-24T10:22:37.573 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T10:22:37.581 INFO:teuthology.task.internal:Opening connections... 2024-03-24T10:22:37.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:22:37.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:37.684 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:22:37.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:37.787 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T10:22:37.794 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-24T10:22:37.810 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-24T10:22:37.811 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-24T10:22:37.867 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-03-24T10:22:37.867 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:22:37.868 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:22:37.869 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-24T10:22:37.892 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-24T10:22:37.909 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-24T10:22:37.910 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-24T10:22:37.966 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-24T10:22:37.966 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:22:37.967 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:22:37.967 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-24T10:22:37.992 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T10:22:38.000 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T10:22:38.014 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T10:22:38.014 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:22:38.016 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:22:38.033 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T10:22:38.039 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T10:22:38.039 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:22:38.073 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:22:38.097 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T10:22:38.151 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T10:22:38.164 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T10:22:38.164 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T10:22:38.165 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:38.165 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:38.182 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-24T10:22:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:38.183 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-24T10:22:38.239 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:22:38.239 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-361.el9.x86_64 2024-03-24T10:22:38.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-03-24T10:22:38.240 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:22:38.240 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-24T10:22:38.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-24T10:22:40.278 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T10:22:40.373 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T10:22:41.712 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-24T10:22:41.823 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T10:22:42.891 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T10:22:44.915 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-24T10:22:46.460 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-24T10:22:48.437 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-24T10:22:49.740 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-03-24T10:22:51.425 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-24T10:22:52.820 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-24T10:22:54.292 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-24T10:22:56.756 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-03-24T10:23:01.981 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-24T10:23:02.778 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 21 MB 00:11 2024-03-24T10:23:04.146 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:23:04.146 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:23:04.245 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-24T10:23:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-24T10:23:04.255 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-24T10:23:05.074 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 64 kB/s | 44 kB 00:00 2024-03-24T10:23:14.033 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 277 B/s | 1.7 kB 00:06 2024-03-24T10:23:16.207 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:23:16.207 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:23:16.293 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-24T10:23:16.298 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-24T10:23:16.299 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-03-24T10:23:16.299 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:23:16.299 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 76 M 2024-03-24T10:23:16.299 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 123 M 2024-03-24T10:23:16.299 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-24T10:23:16.659 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 481 kB/s | 44 kB 00:00 2024-03-24T10:23:17.018 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 33 MB/s | 15 MB 00:00 2024-03-24T10:23:17.435 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-24T10:23:17.743 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-24T10:23:17.744 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:23:17.744 INFO:teuthology.orchestra.run.smithi001.stdout:Total 52 MB/s | 76 MB 00:01 2024-03-24T10:23:18.441 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-24T10:23:18.887 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-24T10:23:18.888 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-24T10:23:19.955 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-24T10:23:19.956 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-24T10:23:20.439 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 2.1 MB/s | 33 MB 00:16 2024-03-24T10:23:20.589 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 942 kB/s | 15 MB 00:16 2024-03-24T10:23:22.554 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-24T10:23:23.223 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:23:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:26.214 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:31.459 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:31.497 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:23:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:23:41.545 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 771 kB/s | 27 MB 00:36 2024-03-24T10:23:41.546 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:23:41.546 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.0 MB/s | 76 MB 00:37 2024-03-24T10:23:42.236 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-24T10:23:42.675 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-24T10:23:42.675 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-24T10:23:43.713 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-24T10:23:43.714 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-24T10:23:46.314 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-24T10:23:46.943 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:23:47.055 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:23:50.082 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:55.179 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:23:55.224 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:00.352 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:07.309 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:07.309 INFO:teuthology.orchestra.run.smithi001.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:24:07.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:24:07.356 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:07.715 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:07.716 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:24:07.716 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:07.716 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:08.134 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:24:08.135 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-24T10:24:08.461 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:08.568 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:08.568 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:24:08.568 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:24:08.569 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:24:08.569 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T10:24:08.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:24:08.569 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:08.651 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:08.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:08.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-03-24T10:24:08.689 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:24:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:24:08.691 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:24:08.691 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-03-24T10:24:08.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:24:08.691 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:24:08.692 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:24:08.692 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:24:08.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:24:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-24T10:24:10.723 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:24:10.764 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-24T10:24:10.766 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-24T10:24:10.781 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.NErF3coMXp 2024-03-24T10:24:10.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NErF3coMXp 2024-03-24T10:24:10.853 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.NErF3coMXp 2024-03-24T10:24:10.997 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.NErF3coMXp is 6KB 2024-03-24T10:24:11.018 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.NErF3coMXp 2024-03-24T10:24:11.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:24:11.104 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:24:11.105 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:24:11.105 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:24:11.105 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:24:11.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:24:11.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-24T10:24:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:31.090 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:24:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:24:31.135 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:31.569 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:31.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:24:31.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:31.569 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:31.985 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:31.985 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:24:31.985 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-24T10:24:31.985 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:31.985 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:31.986 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:24:31.986 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:24:31.986 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:24:31.986 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-24T10:24:32.323 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:32.422 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:32.423 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:32.423 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:24:32.423 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:24:32.423 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:24:32.423 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T10:24:32.423 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:24:32.423 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:24:32.506 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:24:32.506 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:24:32.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-24T10:24:32.555 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-24T10:24:32.555 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-24T10:24:32.555 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-24T10:24:32.556 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:24:32.557 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:24:32.558 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:24:32.558 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:24:32.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:24:33.152 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-24T10:24:34.542 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:24:34.575 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-24T10:24:34.577 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-24T10:24:34.592 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.q9Rs73HUq0 2024-03-24T10:24:34.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.q9Rs73HUq0 2024-03-24T10:24:34.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.q9Rs73HUq0 2024-03-24T10:24:34.805 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.q9Rs73HUq0 is 6KB 2024-03-24T10:24:34.853 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.q9Rs73HUq0 2024-03-24T10:24:34.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:24:34.937 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:24:34.938 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:24:34.938 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:24:34.938 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:24:34.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:24:35.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-24T10:24:41.257 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T10:24:41.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-24T10:24:41.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:05.119 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T10:25:05.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-24T10:25:05.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:41.294 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:25:50.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-24T10:25:50.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:50.604 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-24T10:25:50.983 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-24T10:25:50.984 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:25:50.984 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-24T10:25:50.998 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:25:50.998 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:25:50.998 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:25:50.998 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:25:51.999 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T10:25:52.000 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:25:52.043 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-03-24T10:25:52.066 DEBUG:teuthology.parallel:result is None 2024-03-24T10:26:05.123 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:26:14.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-24T10:26:14.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:14.388 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-24T10:26:14.849 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-24T10:26:14.850 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:26:14.850 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-24T10:26:14.865 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:26:14.865 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:26:14.865 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:26:14.866 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:26:15.867 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T10:26:15.867 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:26:15.927 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-24T10:26:15.971 DEBUG:teuthology.parallel:result is None 2024-03-24T10:26:15.971 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T10:26:15.978 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T10:26:15.979 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:26:15.981 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:26:15.998 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T10:26:16.005 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T10:26:16.056 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T10:26:16.057 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:26:16.059 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:26:16.139 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T10:26:16.146 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T10:26:16.146 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:26:16.148 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:26:16.176 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:26:16.190 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:26:16.191 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:26:16.204 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:26:16.206 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T10:26:16.213 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T10:26:16.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:26:16.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:26:16.274 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T10:26:16.284 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T10:26:16.284 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:26:16.304 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:26:16.329 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:26:16.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:26:16.453 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:16.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:16.579 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:26:16.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:26:16.648 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:26:16.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:26:16.734 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:16.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:26:16.856 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:26:16.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:26:16.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-24T10:26:16.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-24T10:26:16.956 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:26:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:26:17.373 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T10:26:17.381 INFO:teuthology.task.internal:Starting timer... 2024-03-24T10:26:17.382 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T10:26:17.393 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T10:26:17.402 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-24T10:26:17.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-03-24T10:26:17.473 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-03-24T10:26:17.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-24T10:26:17.533 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-24T10:26:17.535 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T10:26:17.535 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-03-24T10:26:17.559 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-03-24T10:26:17.559 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-24T10:26:17.590 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-24T10:26:17.590 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi003.front.sepia.ceph.com': 'permissive'} 2024-03-24T10:26:17.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:26:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:26:17.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:26:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:26:17.658 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T10:26:17.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:26:17.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:26:17.742 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T10:26:17.752 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T10:26:17.768 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T10:26:17.769 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi003.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T10:30:39.296 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi003.front.sepia.ceph.com')] 2024-03-24T10:30:39.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-24T10:30:39.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:39.438 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-24T10:30:39.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-24T10:30:39.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-24T10:30:39.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:39.622 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-24T10:30:39.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-24T10:30:39.704 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T10:30:39.721 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T10:30:39.722 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:30:39.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:30:39.725 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:30:39.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:30:39.754 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:30:39.771 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:30:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:30:39.798 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:30:39.800 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-03-24T10:30:39.814 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-03-24T10:30:39.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-24T10:30:39.829 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:30:39.842 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-24T10:30:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:30:39.859 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:30:39.876 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:30:39.903 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.904 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.935 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.938 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:30:39.939 INFO:teuthology.run_tasks:Running task install... 2024-03-24T10:30:39.948 DEBUG:teuthology.task.install:project ceph 2024-03-24T10:30:39.948 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T10:30:39.948 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T10:30:39.948 INFO:teuthology.task.install:Using flavor: default 2024-03-24T10:30:39.954 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T10:30:39.955 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T10:30:39.955 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:30:39.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:39.957 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:30:39.957 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:40.174 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:30:40.174 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:30:40.179 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:30:40.179 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:30:40.295 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:30:40.295 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:30:40.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:30:40.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:30:40.325 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:30:40.353 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:30:40.356 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:30:40.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:30:40.386 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:30:40.386 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:30:40.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:30:40.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:30:40.474 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:30:40.537 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:30:40.540 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-03-24T10:30:40.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-03-24T10:30:40.604 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-03-24T10:30:40.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-03-24T10:30:40.786 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-03-24T10:30:40.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:30:40.844 INFO:teuthology.orchestra.run.smithi001.stdout:74 files removed 2024-03-24T10:30:40.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:30:41.506 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 252 kB/s | 83 kB 00:00 2024-03-24T10:30:41.577 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 182 kB/s | 83 kB 00:00 2024-03-24T10:30:41.945 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-24T10:30:41.948 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T10:30:42.445 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-24T10:30:42.456 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-24T10:30:42.890 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T10:30:42.900 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T10:30:44.419 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-24T10:30:44.473 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-24T10:30:47.413 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-03-24T10:30:47.676 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-24T10:30:50.573 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 19 MB 00:02 2024-03-24T10:30:54.924 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-24T10:30:56.724 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-03-24T10:30:58.751 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T10:31:03.926 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-24T10:31:06.390 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:31:06.391 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:31:06.406 INFO:teuthology.orchestra.run.smithi003.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:31:06.407 INFO:teuthology.orchestra.run.smithi003.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:31:06.408 INFO:teuthology.orchestra.run.smithi003.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:31:06.409 INFO:teuthology.orchestra.run.smithi003.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:31:06.411 INFO:teuthology.orchestra.run.smithi003.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:31:06.414 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:31:06.415 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:31:06.502 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:31:06.524 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:31:06.525 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:31:06.526 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:31:06.527 INFO:teuthology.orchestra.run.smithi003.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:31:06.528 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:31:06.529 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:31:06.530 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:31:06.531 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:31:06.532 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:31:06.533 INFO:teuthology.orchestra.run.smithi003.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:31:06.534 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:31:06.535 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:31:06.536 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:31:06.536 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:31:06.536 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:31:06.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:31:06.539 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:31:06.540 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:31:06.540 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:31:06.540 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:31:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout:Install 150 Packages 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 24 Packages 2024-03-24T10:31:06.543 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:31:06.544 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 224 M 2024-03-24T10:31:06.544 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-24T10:31:07.185 INFO:teuthology.orchestra.run.smithi003.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-24T10:31:07.544 INFO:teuthology.orchestra.run.smithi003.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.4 MB/s | 888 kB 00:00 2024-03-24T10:31:07.636 INFO:teuthology.orchestra.run.smithi003.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.5 MB/s | 145 kB 00:00 2024-03-24T10:31:07.761 INFO:teuthology.orchestra.run.smithi003.stdout:(4/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 6.8 MB/s | 5.3 MB 00:00 2024-03-24T10:31:07.962 INFO:teuthology.orchestra.run.smithi003.stdout:(5/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 7.2 MB/s | 2.4 MB 00:00 2024-03-24T10:31:07.995 INFO:teuthology.orchestra.run.smithi003.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-03-24T10:31:08.596 INFO:teuthology.orchestra.run.smithi003.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 7.4 MB/s | 4.7 MB 00:00 2024-03-24T10:31:09.229 INFO:teuthology.orchestra.run.smithi003.stdout:(8/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 9.1 MB/s | 21 MB 00:02 2024-03-24T10:31:09.321 INFO:teuthology.orchestra.run.smithi003.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 275 kB/s | 25 kB 00:00 2024-03-24T10:31:09.880 INFO:teuthology.orchestra.run.smithi003.stdout:(10/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 8.9 MB/s | 17 MB 00:01 2024-03-24T10:31:09.980 INFO:teuthology.orchestra.run.smithi003.stdout:(11/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 7.1 MB/s | 9.8 MB 00:01 2024-03-24T10:31:10.005 INFO:teuthology.orchestra.run.smithi003.stdout:(12/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 219 kB 00:00 2024-03-24T10:31:10.038 INFO:teuthology.orchestra.run.smithi003.stdout:(13/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 463 kB/s | 27 kB 00:00 2024-03-24T10:31:10.055 INFO:teuthology.orchestra.run.smithi003.stdout:(14/174): libcephfs-devel-19.0.0-1855.g21b3bd67 631 kB/s | 32 kB 00:00 2024-03-24T10:31:10.147 INFO:teuthology.orchestra.run.smithi003.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 163 kB 00:00 2024-03-24T10:31:10.166 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 903 kB/s | 19 MB 00:21 2024-03-24T10:31:10.190 INFO:teuthology.orchestra.run.smithi003.stdout:(16/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 4.8 MB/s | 744 kB 00:00 2024-03-24T10:31:10.214 INFO:teuthology.orchestra.run.smithi003.stdout:(17/174): librados-devel-19.0.0-1855.g21b3bd67. 1.9 MB/s | 127 kB 00:00 2024-03-24T10:31:10.290 INFO:teuthology.orchestra.run.smithi003.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 4.9 MB/s | 499 kB 00:00 2024-03-24T10:31:10.382 INFO:teuthology.orchestra.run.smithi003.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 492 kB/s | 45 kB 00:00 2024-03-24T10:31:10.465 INFO:teuthology.orchestra.run.smithi003.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.5 MB/s | 131 kB 00:00 2024-03-24T10:31:10.566 INFO:teuthology.orchestra.run.smithi003.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.6 MB/s | 161 kB 00:00 2024-03-24T10:31:10.675 INFO:teuthology.orchestra.run.smithi003.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.9 MB/s | 320 kB 00:00 2024-03-24T10:31:10.774 INFO:teuthology.orchestra.run.smithi003.stdout:(23/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 9.2 MB/s | 5.1 MB 00:00 2024-03-24T10:31:10.800 INFO:teuthology.orchestra.run.smithi003.stdout:(24/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.3 MB/s | 298 kB 00:00 2024-03-24T10:31:10.863 INFO:teuthology.orchestra.run.smithi003.stdout:(25/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.2 MB/s | 99 kB 00:00 2024-03-24T10:31:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:(26/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-24T10:31:10.992 INFO:teuthology.orchestra.run.smithi003.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-24T10:31:11.084 INFO:teuthology.orchestra.run.smithi003.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 343 kB/s | 31 kB 00:00 2024-03-24T10:31:11.210 INFO:teuthology.orchestra.run.smithi003.stdout:(29/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 8.8 MB/s | 3.1 MB 00:00 2024-03-24T10:31:11.235 INFO:teuthology.orchestra.run.smithi003.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 919 kB/s | 138 kB 00:00 2024-03-24T10:31:11.587 INFO:teuthology.orchestra.run.smithi003.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 4.8 MB/s | 1.8 MB 00:00 2024-03-24T10:31:11.685 INFO:teuthology.orchestra.run.smithi003.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.4 MB/s | 248 kB 00:00 2024-03-24T10:31:11.761 INFO:teuthology.orchestra.run.smithi003.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 657 kB/s | 49 kB 00:00 2024-03-24T10:31:11.836 INFO:teuthology.orchestra.run.smithi003.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 205 kB/s | 15 kB 00:00 2024-03-24T10:31:11.938 INFO:teuthology.orchestra.run.smithi003.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.6 MB/s | 264 kB 00:00 2024-03-24T10:31:12.103 INFO:teuthology.orchestra.run.smithi003.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-24T10:31:12.187 INFO:teuthology.orchestra.run.smithi003.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 219 kB/s | 18 kB 00:00 2024-03-24T10:31:12.287 INFO:teuthology.orchestra.run.smithi003.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-03-24T10:31:12.438 INFO:teuthology.orchestra.run.smithi003.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-24T10:31:12.479 INFO:teuthology.orchestra.run.smithi003.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2024-03-24T10:31:12.505 INFO:teuthology.orchestra.run.smithi003.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T10:31:12.530 INFO:teuthology.orchestra.run.smithi003.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T10:31:12.555 INFO:teuthology.orchestra.run.smithi003.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-24T10:31:12.993 INFO:teuthology.orchestra.run.smithi003.stdout:(44/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 13 MB/s | 47 MB 00:03 2024-03-24T10:31:13.048 INFO:teuthology.orchestra.run.smithi003.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.8 MB/s | 3.0 MB 00:00 2024-03-24T10:31:13.073 INFO:teuthology.orchestra.run.smithi003.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 33 kB/s | 17 kB 00:00 2024-03-24T10:31:13.098 INFO:teuthology.orchestra.run.smithi003.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 10 kB 00:00 2024-03-24T10:31:13.123 INFO:teuthology.orchestra.run.smithi003.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-24T10:31:13.149 INFO:teuthology.orchestra.run.smithi003.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-24T10:31:13.190 INFO:teuthology.orchestra.run.smithi003.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-24T10:31:13.216 INFO:teuthology.orchestra.run.smithi003.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-24T10:31:13.241 INFO:teuthology.orchestra.run.smithi003.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-03-24T10:31:13.266 INFO:teuthology.orchestra.run.smithi003.stdout:(53/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 623 kB/s | 46 kB 00:00 2024-03-24T10:31:13.291 INFO:teuthology.orchestra.run.smithi003.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-24T10:31:13.316 INFO:teuthology.orchestra.run.smithi003.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-24T10:31:13.342 INFO:teuthology.orchestra.run.smithi003.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-24T10:31:13.584 INFO:teuthology.orchestra.run.smithi003.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-03-24T10:31:13.618 INFO:teuthology.orchestra.run.smithi003.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 2.9 MB/s | 811 kB 00:00 2024-03-24T10:31:13.651 INFO:teuthology.orchestra.run.smithi003.stdout:(59/174): postgresql-13.11-1.el9.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2024-03-24T10:31:13.676 INFO:teuthology.orchestra.run.smithi003.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-24T10:31:13.793 INFO:teuthology.orchestra.run.smithi003.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-24T10:31:13.818 INFO:teuthology.orchestra.run.smithi003.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 562 kB/s | 78 kB 00:00 2024-03-24T10:31:13.852 INFO:teuthology.orchestra.run.smithi003.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.9 MB/s | 1.0 MB 00:00 2024-03-24T10:31:13.944 INFO:teuthology.orchestra.run.smithi003.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-24T10:31:13.969 INFO:teuthology.orchestra.run.smithi003.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 169 kB/s | 24 kB 00:00 2024-03-24T10:31:13.995 INFO:teuthology.orchestra.run.smithi003.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-03-24T10:31:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T10:31:14.045 INFO:teuthology.orchestra.run.smithi003.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 637 kB/s | 32 kB 00:00 2024-03-24T10:31:14.137 INFO:teuthology.orchestra.run.smithi003.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-24T10:31:14.162 INFO:teuthology.orchestra.run.smithi003.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-24T10:31:14.187 INFO:teuthology.orchestra.run.smithi003.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-24T10:31:14.204 INFO:teuthology.orchestra.run.smithi003.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-24T10:31:14.230 INFO:teuthology.orchestra.run.smithi003.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 988 kB/s | 41 kB 00:00 2024-03-24T10:31:14.255 INFO:teuthology.orchestra.run.smithi003.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T10:31:14.280 INFO:teuthology.orchestra.run.smithi003.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T10:31:14.305 INFO:teuthology.orchestra.run.smithi003.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T10:31:14.322 INFO:teuthology.orchestra.run.smithi003.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-03-24T10:31:14.339 INFO:teuthology.orchestra.run.smithi003.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 737 kB/s | 43 kB 00:00 2024-03-24T10:31:14.366 INFO:teuthology.orchestra.run.smithi003.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 929 kB/s | 39 kB 00:00 2024-03-24T10:31:14.531 INFO:teuthology.orchestra.run.smithi003.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 213 kB/s | 41 kB 00:00 2024-03-24T10:31:14.665 INFO:teuthology.orchestra.run.smithi003.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 300 kB 00:00 2024-03-24T10:31:14.786 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-24T10:31:14.815 INFO:teuthology.orchestra.run.smithi003.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-24T10:31:14.832 INFO:teuthology.orchestra.run.smithi003.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 189 kB/s | 56 kB 00:00 2024-03-24T10:31:14.849 INFO:teuthology.orchestra.run.smithi003.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 344 kB/s | 63 kB 00:00 2024-03-24T10:31:14.883 INFO:teuthology.orchestra.run.smithi003.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T10:31:14.899 INFO:teuthology.orchestra.run.smithi003.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-03-24T10:31:14.925 INFO:teuthology.orchestra.run.smithi003.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-24T10:31:14.967 INFO:teuthology.orchestra.run.smithi003.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T10:31:14.992 INFO:teuthology.orchestra.run.smithi003.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2024-03-24T10:31:15.017 INFO:teuthology.orchestra.run.smithi003.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-24T10:31:15.042 INFO:teuthology.orchestra.run.smithi003.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-24T10:31:15.167 INFO:teuthology.orchestra.run.smithi003.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-24T10:31:15.193 INFO:teuthology.orchestra.run.smithi003.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-24T10:31:15.218 INFO:teuthology.orchestra.run.smithi003.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-24T10:31:15.252 INFO:teuthology.orchestra.run.smithi003.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-24T10:31:15.285 INFO:teuthology.orchestra.run.smithi003.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-24T10:31:15.319 INFO:teuthology.orchestra.run.smithi003.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-03-24T10:31:15.394 INFO:teuthology.orchestra.run.smithi003.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-24T10:31:15.445 INFO:teuthology.orchestra.run.smithi003.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-24T10:31:15.472 INFO:teuthology.orchestra.run.smithi003.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 216 kB 00:00 2024-03-24T10:31:15.495 INFO:teuthology.orchestra.run.smithi003.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 696 kB/s | 35 kB 00:00 2024-03-24T10:31:15.521 INFO:teuthology.orchestra.run.smithi003.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-24T10:31:15.546 INFO:teuthology.orchestra.run.smithi003.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-24T10:31:15.588 INFO:teuthology.orchestra.run.smithi003.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-03-24T10:31:15.654 INFO:teuthology.orchestra.run.smithi003.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-24T10:31:15.721 INFO:teuthology.orchestra.run.smithi003.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-03-24T10:31:16.089 INFO:teuthology.orchestra.run.smithi003.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 53 kB/s | 19 kB 00:00 2024-03-24T10:31:16.242 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-03-24T10:31:16.272 INFO:teuthology.orchestra.run.smithi003.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 801 kB/s | 548 kB 00:00 2024-03-24T10:31:16.306 INFO:teuthology.orchestra.run.smithi003.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 475 kB/s | 309 kB 00:00 2024-03-24T10:31:16.372 INFO:teuthology.orchestra.run.smithi003.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-24T10:31:16.414 INFO:teuthology.orchestra.run.smithi003.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-24T10:31:16.473 INFO:teuthology.orchestra.run.smithi003.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-24T10:31:16.532 INFO:teuthology.orchestra.run.smithi003.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-24T10:31:16.707 INFO:teuthology.orchestra.run.smithi003.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-24T10:31:16.740 INFO:teuthology.orchestra.run.smithi003.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-03-24T10:31:16.824 INFO:teuthology.orchestra.run.smithi003.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-03-24T10:31:16.849 INFO:teuthology.orchestra.run.smithi003.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 306 kB/s | 43 kB 00:00 2024-03-24T10:31:16.875 INFO:teuthology.orchestra.run.smithi003.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-03-24T10:31:16.925 INFO:teuthology.orchestra.run.smithi003.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-24T10:31:16.975 INFO:teuthology.orchestra.run.smithi003.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-24T10:31:17.010 INFO:teuthology.orchestra.run.smithi003.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-24T10:31:17.034 INFO:teuthology.orchestra.run.smithi003.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-24T10:31:17.076 INFO:teuthology.orchestra.run.smithi003.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-24T10:31:17.118 INFO:teuthology.orchestra.run.smithi003.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-24T10:31:17.176 INFO:teuthology.orchestra.run.smithi003.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-24T10:31:17.202 INFO:teuthology.orchestra.run.smithi003.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-24T10:31:17.227 INFO:teuthology.orchestra.run.smithi003.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-24T10:31:17.277 INFO:teuthology.orchestra.run.smithi003.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-24T10:31:17.302 INFO:teuthology.orchestra.run.smithi003.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-24T10:31:17.336 INFO:teuthology.orchestra.run.smithi003.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-03-24T10:31:17.370 INFO:teuthology.orchestra.run.smithi003.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-24T10:31:17.428 INFO:teuthology.orchestra.run.smithi003.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-03-24T10:31:17.462 INFO:teuthology.orchestra.run.smithi003.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-24T10:31:17.487 INFO:teuthology.orchestra.run.smithi003.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-03-24T10:31:17.521 INFO:teuthology.orchestra.run.smithi003.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2024-03-24T10:31:17.573 INFO:teuthology.orchestra.run.smithi003.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-24T10:31:17.588 INFO:teuthology.orchestra.run.smithi003.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-24T10:31:17.622 INFO:teuthology.orchestra.run.smithi003.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-03-24T10:31:17.672 INFO:teuthology.orchestra.run.smithi003.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-24T10:31:17.697 INFO:teuthology.orchestra.run.smithi003.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-24T10:31:17.734 INFO:teuthology.orchestra.run.smithi003.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-24T10:31:17.776 INFO:teuthology.orchestra.run.smithi003.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-24T10:31:17.798 INFO:teuthology.orchestra.run.smithi003.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-03-24T10:31:17.832 INFO:teuthology.orchestra.run.smithi003.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-24T10:31:17.882 INFO:teuthology.orchestra.run.smithi003.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T10:31:17.907 INFO:teuthology.orchestra.run.smithi003.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 826 kB/s | 90 kB 00:00 2024-03-24T10:31:17.958 INFO:teuthology.orchestra.run.smithi003.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-24T10:31:17.983 INFO:teuthology.orchestra.run.smithi003.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 198 kB/s | 20 kB 00:00 2024-03-24T10:31:18.016 INFO:teuthology.orchestra.run.smithi003.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-24T10:31:18.167 INFO:teuthology.orchestra.run.smithi003.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-24T10:31:18.202 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T10:31:18.208 INFO:teuthology.orchestra.run.smithi003.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T10:31:18.334 INFO:teuthology.orchestra.run.smithi003.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 19 MB/s | 2.3 MB 00:00 2024-03-24T10:31:18.409 INFO:teuthology.orchestra.run.smithi003.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 25 MB/s | 1.8 MB 00:00 2024-03-24T10:31:18.451 INFO:teuthology.orchestra.run.smithi003.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-24T10:31:18.493 INFO:teuthology.orchestra.run.smithi003.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.9 MB/s | 248 kB 00:00 2024-03-24T10:31:18.535 INFO:teuthology.orchestra.run.smithi003.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-24T10:31:18.577 INFO:teuthology.orchestra.run.smithi003.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 876 kB/s | 36 kB 00:00 2024-03-24T10:31:18.627 INFO:teuthology.orchestra.run.smithi003.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-24T10:31:18.669 INFO:teuthology.orchestra.run.smithi003.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-24T10:31:18.719 INFO:teuthology.orchestra.run.smithi003.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 459 kB/s | 23 kB 00:00 2024-03-24T10:31:18.761 INFO:teuthology.orchestra.run.smithi003.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-24T10:31:18.803 INFO:teuthology.orchestra.run.smithi003.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-24T10:31:18.845 INFO:teuthology.orchestra.run.smithi003.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-24T10:31:18.896 INFO:teuthology.orchestra.run.smithi003.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.3 MB/s | 64 kB 00:00 2024-03-24T10:31:18.938 INFO:teuthology.orchestra.run.smithi003.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 693 kB/s | 29 kB 00:00 2024-03-24T10:31:18.988 INFO:teuthology.orchestra.run.smithi003.stdout:(166/174): librados2-19.0.0-1855.g21b3bd67.el9. 3.4 MB/s | 3.4 MB 00:01 2024-03-24T10:31:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-24T10:31:19.113 INFO:teuthology.orchestra.run.smithi003.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 25 MB/s | 2.3 MB 00:00 2024-03-24T10:31:19.155 INFO:teuthology.orchestra.run.smithi003.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 13 MB/s | 2.2 MB 00:00 2024-03-24T10:31:19.181 INFO:teuthology.orchestra.run.smithi003.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-24T10:31:19.206 INFO:teuthology.orchestra.run.smithi003.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 620 kB/s | 31 kB 00:00 2024-03-24T10:31:19.240 INFO:teuthology.orchestra.run.smithi003.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-24T10:31:19.290 INFO:teuthology.orchestra.run.smithi003.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 890 kB/s | 74 kB 00:00 2024-03-24T10:31:20.415 INFO:teuthology.orchestra.run.smithi003.stdout:(174/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-03-24T10:31:20.421 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:20.421 INFO:teuthology.orchestra.run.smithi003.stdout:Total 16 MB/s | 224 MB 00:13 2024-03-24T10:31:20.900 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-24T10:31:21.199 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-24T10:31:21.199 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-24T10:31:23.292 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-24T10:31:23.292 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-24T10:31:23.352 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-24T10:31:25.835 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:31:25.836 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:31:25.852 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:31:25.852 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:31:25.853 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:31:25.854 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:31:25.857 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:31:25.860 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:31:25.861 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:31:25.947 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-24T10:31:25.964 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:31:25.965 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:31:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:31:25.967 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:31:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:31:25.969 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:31:25.970 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:31:25.971 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:31:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:31:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:31:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:31:25.972 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:31:25.973 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:31:25.974 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:31:25.975 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:31:25.976 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:31:25.977 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:31:25.978 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:Install 150 Packages 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 24 Packages 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:31:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 224 M 2024-03-24T10:31:25.980 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-24T10:31:26.654 INFO:teuthology.orchestra.run.smithi001.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-24T10:31:27.363 INFO:teuthology.orchestra.run.smithi001.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.2 MB/s | 888 kB 00:00 2024-03-24T10:31:27.454 INFO:teuthology.orchestra.run.smithi001.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.5 MB/s | 145 kB 00:00 2024-03-24T10:31:28.229 INFO:teuthology.orchestra.run.smithi001.stdout:(4/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 2.9 MB/s | 5.3 MB 00:01 2024-03-24T10:31:28.271 INFO:teuthology.orchestra.run.smithi001.stdout:(5/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-03-24T10:31:28.596 INFO:teuthology.orchestra.run.smithi001.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 4.1 MB/s | 1.5 MB 00:00 2024-03-24T10:31:28.989 INFO:teuthology.orchestra.run.smithi001.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 6.5 MB/s | 4.7 MB 00:00 2024-03-24T10:31:30.414 INFO:teuthology.orchestra.run.smithi001.stdout:(8/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el9 6.9 MB/s | 9.8 MB 00:01 2024-03-24T10:31:30.489 INFO:teuthology.orchestra.run.smithi001.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 336 kB/s | 25 kB 00:00 2024-03-24T10:31:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-24T10:31:30.644 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T10:31:30.731 INFO:teuthology.orchestra.run.smithi001.stdout:(10/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 7.9 MB/s | 17 MB 00:02 2024-03-24T10:31:30.769 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T10:31:30.815 INFO:teuthology.orchestra.run.smithi001.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.6 MB/s | 219 kB 00:00 2024-03-24T10:31:30.882 INFO:teuthology.orchestra.run.smithi001.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 405 kB/s | 27 kB 00:00 2024-03-24T10:31:30.900 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:31:30.911 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:31:30.949 INFO:teuthology.orchestra.run.smithi001.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 477 kB/s | 32 kB 00:00 2024-03-24T10:31:30.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T10:31:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T10:31:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T10:31:31.090 INFO:teuthology.orchestra.run.smithi001.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 5.1 MB/s | 744 kB 00:00 2024-03-24T10:31:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T10:31:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T10:31:31.166 INFO:teuthology.orchestra.run.smithi001.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 2.1 MB/s | 163 kB 00:00 2024-03-24T10:31:31.233 INFO:teuthology.orchestra.run.smithi001.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.9 MB/s | 127 kB 00:00 2024-03-24T10:31:31.325 INFO:teuthology.orchestra.run.smithi001.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 5.3 MB/s | 499 kB 00:00 2024-03-24T10:31:31.499 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T10:31:31.561 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:31:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:31:31.695 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:31:31.787 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:31:31.825 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:31:31.873 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T10:31:31.892 INFO:teuthology.orchestra.run.smithi001.stdout:(18/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 9.0 MB/s | 5.1 MB 00:00 2024-03-24T10:31:31.959 INFO:teuthology.orchestra.run.smithi001.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 680 kB/s | 45 kB 00:00 2024-03-24T10:31:32.034 INFO:teuthology.orchestra.run.smithi001.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.7 MB/s | 131 kB 00:00 2024-03-24T10:31:32.051 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T10:31:32.098 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T10:31:32.109 INFO:teuthology.orchestra.run.smithi001.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 2.1 MB/s | 161 kB 00:00 2024-03-24T10:31:32.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:31:32.184 INFO:teuthology.orchestra.run.smithi001.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 4.2 MB/s | 320 kB 00:00 2024-03-24T10:31:32.214 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:31:32.243 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:31:32.269 INFO:teuthology.orchestra.run.smithi001.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.5 MB/s | 298 kB 00:00 2024-03-24T10:31:32.309 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:31:32.339 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T10:31:32.343 INFO:teuthology.orchestra.run.smithi001.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.3 MB/s | 99 kB 00:00 2024-03-24T10:31:32.362 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:31:32.377 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:31:32.418 INFO:teuthology.orchestra.run.smithi001.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-24T10:31:32.583 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:31:32.606 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:31:32.645 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T10:31:32.764 INFO:teuthology.orchestra.run.smithi001.stdout:(26/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 9.0 MB/s | 3.1 MB 00:00 2024-03-24T10:31:32.811 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T10:31:32.836 INFO:teuthology.orchestra.run.smithi001.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.3 MB/s | 170 kB 00:00 2024-03-24T10:31:32.903 INFO:teuthology.orchestra.run.smithi001.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 471 kB/s | 31 kB 00:00 2024-03-24T10:31:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:31:32.970 INFO:teuthology.orchestra.run.smithi001.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 2.0 MB/s | 138 kB 00:00 2024-03-24T10:31:33.008 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:31:33.101 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:31:33.183 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:31:33.187 INFO:teuthology.orchestra.run.smithi001.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 8.2 MB/s | 1.8 MB 00:00 2024-03-24T10:31:33.407 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:31:33.517 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T10:31:33.621 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T10:31:33.655 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T10:31:33.682 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T10:31:33.726 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T10:31:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T10:31:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T10:31:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T10:31:33.970 INFO:teuthology.orchestra.run.smithi001.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 9.4 MB/s | 7.4 MB 00:00 2024-03-24T10:31:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T10:31:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T10:31:34.054 INFO:teuthology.orchestra.run.smithi001.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.9 MB/s | 248 kB 00:00 2024-03-24T10:31:34.067 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T10:31:34.101 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T10:31:34.121 INFO:teuthology.orchestra.run.smithi001.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 740 kB/s | 49 kB 00:00 2024-03-24T10:31:34.127 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T10:31:34.188 INFO:teuthology.orchestra.run.smithi001.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 229 kB/s | 15 kB 00:00 2024-03-24T10:31:34.200 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T10:31:34.233 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T10:31:34.263 INFO:teuthology.orchestra.run.smithi001.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 3.4 MB/s | 264 kB 00:00 2024-03-24T10:31:34.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T10:31:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T10:31:34.363 INFO:teuthology.orchestra.run.smithi001.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-24T10:31:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T10:31:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T10:31:34.422 INFO:teuthology.orchestra.run.smithi001.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 313 kB/s | 18 kB 00:00 2024-03-24T10:31:34.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T10:31:34.572 INFO:teuthology.orchestra.run.smithi001.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-24T10:31:34.631 INFO:teuthology.orchestra.run.smithi001.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-24T10:31:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T10:31:34.665 INFO:teuthology.orchestra.run.smithi001.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-24T10:31:34.689 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T10:31:34.706 INFO:teuthology.orchestra.run.smithi001.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-24T10:31:34.740 INFO:teuthology.orchestra.run.smithi001.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 949 kB/s | 32 kB 00:00 2024-03-24T10:31:34.832 INFO:teuthology.orchestra.run.smithi001.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-24T10:31:34.866 INFO:teuthology.orchestra.run.smithi001.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-24T10:31:34.899 INFO:teuthology.orchestra.run.smithi001.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-24T10:31:34.933 INFO:teuthology.orchestra.run.smithi001.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-24T10:31:34.970 INFO:teuthology.orchestra.run.smithi001.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:31:35.009 INFO:teuthology.orchestra.run.smithi001.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-24T10:31:35.051 INFO:teuthology.orchestra.run.smithi001.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-24T10:31:35.084 INFO:teuthology.orchestra.run.smithi001.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-24T10:31:35.126 INFO:teuthology.orchestra.run.smithi001.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-24T10:31:35.160 INFO:teuthology.orchestra.run.smithi001.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-24T10:31:35.193 INFO:teuthology.orchestra.run.smithi001.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2024-03-24T10:31:35.227 INFO:teuthology.orchestra.run.smithi001.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-24T10:31:35.255 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T10:31:35.336 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T10:31:35.352 INFO:teuthology.orchestra.run.smithi001.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-24T10:31:35.419 INFO:teuthology.orchestra.run.smithi001.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-24T10:31:35.427 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T10:31:35.461 INFO:teuthology.orchestra.run.smithi001.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 19 MB/s | 811 kB 00:00 2024-03-24T10:31:35.478 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:31:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:31:35.495 INFO:teuthology.orchestra.run.smithi001.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-24T10:31:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T10:31:35.593 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T10:31:35.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T10:31:35.660 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T10:31:35.745 INFO:teuthology.orchestra.run.smithi001.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-24T10:31:35.795 INFO:teuthology.orchestra.run.smithi001.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-24T10:31:35.829 INFO:teuthology.orchestra.run.smithi001.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-24T10:31:35.951 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T10:31:35.988 INFO:teuthology.orchestra.run.smithi001.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-03-24T10:31:36.021 INFO:teuthology.orchestra.run.smithi001.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 733 kB/s | 24 kB 00:00 2024-03-24T10:31:36.058 INFO:teuthology.orchestra.run.smithi001.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 546 kB/s | 18 kB 00:00 2024-03-24T10:31:36.097 INFO:teuthology.orchestra.run.smithi001.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T10:31:36.139 INFO:teuthology.orchestra.run.smithi001.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 759 kB/s | 32 kB 00:00 2024-03-24T10:31:36.248 INFO:teuthology.orchestra.run.smithi001.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 4.8 MB 00:00 2024-03-24T10:31:36.282 INFO:teuthology.orchestra.run.smithi001.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-24T10:31:36.299 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T10:31:36.315 INFO:teuthology.orchestra.run.smithi001.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-24T10:31:36.349 INFO:teuthology.orchestra.run.smithi001.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-24T10:31:36.363 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T10:31:36.409 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T10:31:36.562 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T10:31:36.609 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T10:31:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T10:31:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T10:31:36.757 INFO:teuthology.orchestra.run.smithi001.stdout:(71/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 7.6 MB/s | 47 MB 00:06 2024-03-24T10:31:36.774 INFO:teuthology.orchestra.run.smithi001.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 97 kB/s | 41 kB 00:00 2024-03-24T10:31:36.776 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T10:31:36.800 INFO:teuthology.orchestra.run.smithi001.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-24T10:31:36.813 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T10:31:36.825 INFO:teuthology.orchestra.run.smithi001.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T10:31:36.850 INFO:teuthology.orchestra.run.smithi001.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T10:31:36.860 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T10:31:36.875 INFO:teuthology.orchestra.run.smithi001.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-24T10:31:36.901 INFO:teuthology.orchestra.run.smithi001.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-24T10:31:36.903 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T10:31:36.948 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T10:31:36.989 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T10:31:37.076 INFO:teuthology.orchestra.run.smithi001.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 224 kB/s | 39 kB 00:00 2024-03-24T10:31:37.130 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T10:31:37.177 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T10:31:37.207 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T10:31:37.243 INFO:teuthology.orchestra.run.smithi001.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-24T10:31:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T10:31:37.268 INFO:teuthology.orchestra.run.smithi001.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-03-24T10:31:37.293 INFO:teuthology.orchestra.run.smithi001.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T10:31:37.319 INFO:teuthology.orchestra.run.smithi001.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T10:31:37.344 INFO:teuthology.orchestra.run.smithi001.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T10:31:37.355 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T10:31:37.369 INFO:teuthology.orchestra.run.smithi001.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-24T10:31:37.386 INFO:teuthology.orchestra.run.smithi001.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-24T10:31:37.407 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T10:31:37.411 INFO:teuthology.orchestra.run.smithi001.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-24T10:31:37.445 INFO:teuthology.orchestra.run.smithi001.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T10:31:37.447 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T10:31:37.462 INFO:teuthology.orchestra.run.smithi001.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-24T10:31:37.473 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T10:31:37.487 INFO:teuthology.orchestra.run.smithi001.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-24T10:31:37.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T10:31:37.513 INFO:teuthology.orchestra.run.smithi001.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-24T10:31:37.577 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T10:31:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T10:31:37.755 INFO:teuthology.orchestra.run.smithi001.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 703 kB/s | 187 kB 00:00 2024-03-24T10:31:37.788 INFO:teuthology.orchestra.run.smithi001.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-03-24T10:31:37.810 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T10:31:37.838 INFO:teuthology.orchestra.run.smithi001.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-24T10:31:37.856 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T10:31:37.864 INFO:teuthology.orchestra.run.smithi001.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-24T10:31:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T10:31:37.970 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T10:31:38.163 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T10:31:38.392 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T10:31:38.406 INFO:teuthology.orchestra.run.smithi001.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 38 kB/s | 21 kB 00:00 2024-03-24T10:31:38.456 INFO:teuthology.orchestra.run.smithi001.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-03-24T10:31:38.615 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T10:31:38.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T10:31:38.692 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T10:31:38.721 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T10:31:38.748 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T10:31:38.783 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T10:31:38.823 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T10:31:38.863 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T10:31:38.892 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T10:31:38.972 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T10:31:39.011 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T10:31:39.044 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T10:31:39.086 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T10:31:39.120 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T10:31:39.214 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T10:31:39.303 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T10:31:39.351 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T10:31:39.382 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T10:31:39.430 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T10:31:39.431 INFO:teuthology.orchestra.run.smithi001.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 252 kB/s | 245 kB 00:00 2024-03-24T10:31:39.542 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T10:31:39.593 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T10:31:40.366 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T10:31:40.398 INFO:teuthology.orchestra.run.smithi001.stdout:(98/174): ceph-common-19.0.0-1855.g21b3bd67.el9 1.5 MB/s | 21 MB 00:13 2024-03-24T10:31:40.421 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T10:31:40.454 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T10:31:40.504 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T10:31:40.595 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T10:31:40.624 INFO:teuthology.orchestra.run.smithi001.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 208 kB/s | 248 kB 00:01 2024-03-24T10:31:40.629 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T10:31:40.814 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T10:31:40.852 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T10:31:40.932 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T10:31:41.423 INFO:teuthology.orchestra.run.smithi001.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 44 kB/s | 35 kB 00:00 2024-03-24T10:31:41.491 INFO:teuthology.orchestra.run.smithi001.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 198 kB/s | 216 kB 00:01 2024-03-24T10:31:41.615 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T10:31:41.748 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T10:31:42.166 INFO:teuthology.orchestra.run.smithi001.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 60 kB/s | 44 kB 00:00 2024-03-24T10:31:42.300 INFO:teuthology.orchestra.run.smithi001.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 116 kB/s | 94 kB 00:00 2024-03-24T10:31:42.324 INFO:teuthology.orchestra.run.smithi001.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-24T10:31:42.450 INFO:teuthology.orchestra.run.smithi001.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-24T10:31:42.933 INFO:teuthology.orchestra.run.smithi001.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 901 kB/s | 548 kB 00:00 2024-03-24T10:31:42.967 INFO:teuthology.orchestra.run.smithi001.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 598 kB/s | 309 kB 00:00 2024-03-24T10:31:43.042 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T10:31:43.128 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T10:31:43.158 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T10:31:43.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T10:31:43.192 INFO:teuthology.orchestra.run.smithi001.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-24T10:31:43.217 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T10:31:43.459 INFO:teuthology.orchestra.run.smithi001.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-24T10:31:43.546 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T10:31:43.575 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:31:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:31:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T10:31:43.738 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T10:31:43.801 INFO:teuthology.orchestra.run.smithi001.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-24T10:31:43.885 INFO:teuthology.orchestra.run.smithi001.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-03-24T10:31:44.160 INFO:teuthology.orchestra.run.smithi001.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-03-24T10:31:44.219 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T10:31:44.241 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:31:44.329 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:31:44.360 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T10:31:44.377 INFO:teuthology.orchestra.run.smithi001.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2024-03-24T10:31:44.861 INFO:teuthology.orchestra.run.smithi001.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-24T10:31:45.002 INFO:teuthology.orchestra.run.smithi001.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 876 kB/s | 548 kB 00:00 2024-03-24T10:31:45.186 INFO:teuthology.orchestra.run.smithi001.stdout:(116/174): openssl-devel-3.0.7-27.el9.x86_64.rp 624 kB/s | 4.1 MB 00:06 2024-03-24T10:31:45.203 INFO:teuthology.orchestra.run.smithi001.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 127 kB/s | 43 kB 00:00 2024-03-24T10:31:45.263 INFO:teuthology.orchestra.run.smithi001.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 124 kB/s | 32 kB 00:00 2024-03-24T10:31:45.445 INFO:teuthology.orchestra.run.smithi001.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 58 kB/s | 15 kB 00:00 2024-03-24T10:31:45.712 INFO:teuthology.orchestra.run.smithi001.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 337 kB/s | 172 kB 00:00 2024-03-24T10:31:45.848 INFO:teuthology.orchestra.run.smithi001.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 614 kB/s | 358 kB 00:00 2024-03-24T10:31:45.979 INFO:teuthology.orchestra.run.smithi001.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 132 kB/s | 35 kB 00:00 2024-03-24T10:31:46.076 INFO:teuthology.orchestra.run.smithi001.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 480 kB/s | 303 kB 00:00 2024-03-24T10:31:46.430 INFO:teuthology.orchestra.run.smithi001.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 347 kB/s | 203 kB 00:00 2024-03-24T10:31:46.530 INFO:teuthology.orchestra.run.smithi001.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 319 kB/s | 144 kB 00:00 2024-03-24T10:31:46.650 INFO:teuthology.orchestra.run.smithi001.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 49 kB/s | 11 kB 00:00 2024-03-24T10:31:46.718 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:31:46.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:31:46.773 INFO:teuthology.orchestra.run.smithi001.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-03-24T10:31:46.831 INFO:teuthology.orchestra.run.smithi001.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 59 kB/s | 18 kB 00:00 2024-03-24T10:31:46.914 INFO:teuthology.orchestra.run.smithi001.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-03-24T10:31:47.040 INFO:teuthology.orchestra.run.smithi001.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-03-24T10:31:47.099 INFO:teuthology.orchestra.run.smithi001.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-03-24T10:31:47.326 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:31:47.367 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:31:47.390 INFO:teuthology.orchestra.run.smithi001.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-03-24T10:31:47.449 INFO:teuthology.orchestra.run.smithi001.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 225 kB/s | 79 kB 00:00 2024-03-24T10:31:47.460 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:31:47.460 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:31:47.460 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:31:47.460 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:31:47.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:31:47.608 INFO:teuthology.orchestra.run.smithi001.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-24T10:31:47.741 INFO:teuthology.orchestra.run.smithi001.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 165 kB/s | 58 kB 00:00 2024-03-24T10:31:47.875 INFO:teuthology.orchestra.run.smithi001.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-03-24T10:31:47.967 INFO:teuthology.orchestra.run.smithi001.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 527 kB/s | 272 kB 00:00 2024-03-24T10:31:48.084 INFO:teuthology.orchestra.run.smithi001.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 263 kB/s | 90 kB 00:00 2024-03-24T10:31:48.151 INFO:teuthology.orchestra.run.smithi001.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 102 kB/s | 28 kB 00:00 2024-03-24T10:31:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-03-24T10:31:48.501 INFO:teuthology.orchestra.run.smithi001.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-03-24T10:31:48.526 INFO:teuthology.orchestra.run.smithi001.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 122 kB/s | 36 kB 00:00 2024-03-24T10:31:48.552 INFO:teuthology.orchestra.run.smithi001.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 403 kB/s | 188 kB 00:00 2024-03-24T10:31:48.852 INFO:teuthology.orchestra.run.smithi001.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 145 kB/s | 51 kB 00:00 2024-03-24T10:31:48.894 INFO:teuthology.orchestra.run.smithi001.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 262 kB/s | 90 kB 00:00 2024-03-24T10:31:49.036 INFO:teuthology.orchestra.run.smithi001.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-24T10:31:49.162 INFO:teuthology.orchestra.run.smithi001.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 75 kB/s | 20 kB 00:00 2024-03-24T10:31:49.436 INFO:teuthology.orchestra.run.smithi001.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 732 kB/s | 427 kB 00:00 2024-03-24T10:31:49.478 INFO:teuthology.orchestra.run.smithi001.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 433 kB/s | 192 kB 00:00 2024-03-24T10:31:49.903 INFO:teuthology.orchestra.run.smithi001.stdout:(150/174): librados2-19.0.0-1855.g21b3bd67.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-24T10:31:49.945 INFO:teuthology.orchestra.run.smithi001.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-24T10:31:49.970 INFO:teuthology.orchestra.run.smithi001.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-24T10:31:50.021 INFO:teuthology.orchestra.run.smithi001.stdout:(153/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-03-24T10:31:50.221 INFO:teuthology.orchestra.run.smithi001.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 210 kB/s | 42 kB 00:00 2024-03-24T10:31:50.330 INFO:teuthology.orchestra.run.smithi001.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 6.0 MB/s | 2.3 MB 00:00 2024-03-24T10:31:50.430 INFO:teuthology.orchestra.run.smithi001.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.2 MB/s | 248 kB 00:00 2024-03-24T10:31:50.456 INFO:teuthology.orchestra.run.smithi001.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 569 kB/s | 71 kB 00:00 2024-03-24T10:31:50.556 INFO:teuthology.orchestra.run.smithi001.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.1 MB/s | 1.8 MB 00:00 2024-03-24T10:31:50.573 INFO:teuthology.orchestra.run.smithi001.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 256 kB/s | 36 kB 00:00 2024-03-24T10:31:50.598 INFO:teuthology.orchestra.run.smithi001.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.3 MB/s | 192 kB 00:00 2024-03-24T10:31:50.640 INFO:teuthology.orchestra.run.smithi001.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-24T10:31:50.665 INFO:teuthology.orchestra.run.smithi001.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-24T10:31:50.690 INFO:teuthology.orchestra.run.smithi001.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-24T10:31:50.732 INFO:teuthology.orchestra.run.smithi001.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-24T10:31:50.757 INFO:teuthology.orchestra.run.smithi001.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 954 kB/s | 64 kB 00:00 2024-03-24T10:31:50.783 INFO:teuthology.orchestra.run.smithi001.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-24T10:31:50.808 INFO:teuthology.orchestra.run.smithi001.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-24T10:31:50.900 INFO:teuthology.orchestra.run.smithi001.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 19 MB/s | 2.2 MB 00:00 2024-03-24T10:31:50.950 INFO:teuthology.orchestra.run.smithi001.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 1.2 MB 00:00 2024-03-24T10:31:50.984 INFO:teuthology.orchestra.run.smithi001.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 5.4 MB/s | 465 kB 00:00 2024-03-24T10:31:51.026 INFO:teuthology.orchestra.run.smithi001.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-24T10:31:51.084 INFO:teuthology.orchestra.run.smithi001.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-03-24T10:31:51.210 INFO:teuthology.orchestra.run.smithi001.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.7 MB/s | 2.3 MB 00:00 2024-03-24T10:31:51.793 INFO:teuthology.orchestra.run.smithi001.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 97 kB/s | 74 kB 00:00 2024-03-24T10:31:51.800 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:31:51.800 INFO:teuthology.orchestra.run.smithi001.stdout:Total 8.7 MB/s | 224 MB 00:25 2024-03-24T10:31:52.275 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-24T10:31:52.573 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-24T10:31:52.574 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-24T10:31:54.659 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-24T10:31:54.659 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-24T10:31:59.793 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-03-24T10:31:59.794 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:31:59.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:31:59.977 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:31:59.977 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:31:59.977 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:31:59.977 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:01.526 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:32:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:32:01.548 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:01.548 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:32:01.548 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:32:01.549 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:32:01.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:01.791 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:32:01.813 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:32:01.813 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:01.813 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:32:01.813 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:32:01.813 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:32:01.814 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:01.906 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-24T10:32:02.001 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T10:32:02.044 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T10:32:02.074 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:32:02.084 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:32:02.124 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T10:32:02.179 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T10:32:02.219 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T10:32:02.253 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T10:32:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:32:02.297 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T10:32:02.306 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:32:02.307 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:02.307 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:32:02.307 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:32:02.307 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:32:02.307 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T10:32:02.646 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T10:32:02.728 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:32:02.838 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:32:02.853 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:32:02.868 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:32:02.880 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:32:02.900 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T10:32:02.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T10:32:02.971 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:32:02.990 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:32:03.005 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:32:03.045 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T10:32:03.229 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T10:32:03.235 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:32:03.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:32:03.282 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T10:32:03.310 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:32:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:32:03.444 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:32:03.493 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:32:03.540 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T10:32:03.571 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:32:03.587 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:32:03.739 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:32:03.770 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:32:03.818 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T10:32:03.984 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T10:32:04.128 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:32:04.189 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:32:04.275 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:32:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:32:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:32:04.332 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:32:04.416 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:32:04.542 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:32:04.578 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T10:32:04.610 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:32:04.633 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:32:04.634 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:04.634 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:32:04.634 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:32:04.634 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:32:04.634 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:04.658 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T10:32:04.704 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:32:04.715 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:32:04.753 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T10:32:04.767 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T10:32:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T10:32:04.821 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T10:32:04.857 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T10:32:04.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T10:32:04.986 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T10:32:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T10:32:05.081 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T10:32:05.114 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T10:32:05.157 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T10:32:05.190 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T10:32:05.217 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T10:32:05.290 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T10:32:05.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T10:32:05.366 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T10:32:05.395 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T10:32:05.466 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T10:32:05.509 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T10:32:05.586 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T10:32:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T10:32:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:32:05.718 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:32:05.719 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:05.719 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:32:05.719 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:32:05.719 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:32:05.719 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:05.725 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T10:32:05.763 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:32:05.780 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T10:32:05.785 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:32:05.785 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:05.785 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:32:05.785 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:05.831 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:06.165 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:06.187 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:06.333 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T10:32:06.418 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T10:32:06.500 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T10:32:06.562 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:32:06.571 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:32:06.603 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T10:32:06.643 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T10:32:06.681 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T10:32:06.711 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T10:32:07.009 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T10:32:07.313 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T10:32:07.379 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T10:32:07.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T10:32:07.569 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T10:32:07.626 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T10:32:07.683 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T10:32:07.756 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T10:32:07.794 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T10:32:07.830 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T10:32:07.878 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T10:32:07.913 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T10:32:07.956 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T10:32:08.000 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T10:32:08.141 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T10:32:08.187 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T10:32:08.217 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T10:32:08.259 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T10:32:08.357 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T10:32:08.392 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T10:32:08.432 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T10:32:08.457 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T10:32:08.482 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T10:32:08.561 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T10:32:08.698 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T10:32:08.819 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T10:32:08.866 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T10:32:08.924 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T10:32:08.996 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T10:32:09.191 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T10:32:09.245 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T10:32:09.389 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T10:32:09.435 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T10:32:09.466 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T10:32:09.497 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T10:32:09.525 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T10:32:09.559 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T10:32:09.599 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T10:32:09.631 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T10:32:09.660 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T10:32:09.740 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T10:32:09.770 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T10:32:09.804 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T10:32:09.845 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T10:32:09.880 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T10:32:09.974 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T10:32:10.061 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T10:32:10.103 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T10:32:10.134 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T10:32:10.182 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T10:32:10.302 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T10:32:10.539 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T10:32:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:32:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T10:32:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T10:32:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T10:32:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T10:32:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T10:32:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T10:32:11.379 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T10:32:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T10:32:11.440 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T10:32:11.474 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T10:32:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T10:32:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T10:32:11.523 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T10:32:11.554 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T10:32:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T10:32:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T10:32:11.614 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T10:32:11.650 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T10:32:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T10:32:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T10:32:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T10:32:11.834 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T10:32:11.868 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T10:32:11.951 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T10:32:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:12.645 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T10:32:12.782 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T10:32:13.335 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T10:32:13.426 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T10:32:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T10:32:13.550 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T10:32:13.582 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T10:32:13.610 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T10:32:13.633 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T10:32:13.659 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T10:32:13.659 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:32:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:32:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T10:32:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:32:13.878 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:32:13.913 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T10:32:13.945 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T10:32:13.971 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T10:32:13.996 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T10:32:14.030 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T10:32:14.077 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T10:32:14.087 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T10:32:14.108 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T10:32:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T10:32:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T10:32:14.172 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T10:32:14.186 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:32:14.204 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T10:32:14.210 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T10:32:14.216 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:14.216 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:14.235 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T10:32:14.262 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T10:32:14.396 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:14.396 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:32:14.479 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:32:14.503 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:32:14.535 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T10:32:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T10:32:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T10:32:14.590 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T10:32:14.617 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:32:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:32:14.708 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:32:14.739 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T10:32:14.775 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T10:32:15.255 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T10:32:15.282 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:32:15.333 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:32:15.363 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T10:32:17.712 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:32:17.760 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:32:18.335 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:32:18.372 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:32:18.465 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:32:18.465 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:32:18.465 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:32:18.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:18.506 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T10:32:19.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T10:32:19.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T10:32:19.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T10:32:19.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T10:32:19.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T10:32:19.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T10:32:19.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T10:32:19.458 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T10:32:19.459 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T10:32:19.460 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T10:32:19.460 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T10:32:19.461 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T10:32:19.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T10:32:19.463 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T10:32:19.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T10:32:19.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T10:32:19.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T10:32:19.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T10:32:19.464 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T10:32:19.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T10:32:19.466 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T10:32:19.467 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T10:32:19.468 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:32:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T10:32:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T10:32:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T10:32:19.472 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T10:32:19.473 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T10:32:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:32:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:32:22.344 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:32:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:32:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:32:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:32:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:32:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:32:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:32:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:32:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:32:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:32:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:32:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:32:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:22.358 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-24T10:32:22.827 DEBUG:teuthology.parallel:result is None 2024-03-24T10:32:30.831 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-03-24T10:32:30.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:31.012 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:32:31.025 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:32:31.025 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:31.025 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:32:31.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:32.563 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:32:32.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:32.831 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:32:32.854 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:32:32.855 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:32.855 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:32:32.855 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:32:32.855 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:32:32.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:33.323 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:32:33.347 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:32:33.347 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:33.347 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:32:33.347 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:32:33.348 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:32:33.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:33.443 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T10:32:33.810 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:32:33.844 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:32:33.872 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T10:32:33.934 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T10:32:33.963 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:32:34.205 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:32:34.243 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:32:35.263 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:32:35.281 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:32:35.389 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:32:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T10:32:35.590 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:32:35.606 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:32:35.606 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:35.606 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:32:35.606 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:32:35.606 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:32:35.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:32:35.661 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:32:35.717 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T10:32:36.623 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T10:32:36.652 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:32:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:32:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:32:36.742 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:36.742 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:32:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:36.795 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:32:36.818 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:37.138 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:32:37.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:41.876 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:32:41.918 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T10:32:41.947 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T10:32:41.975 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T10:32:42.173 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T10:32:42.295 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T10:32:42.340 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T10:32:42.393 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T10:32:42.439 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T10:32:42.482 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T10:32:42.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T10:32:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T10:32:42.594 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T10:32:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T10:32:42.754 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T10:32:42.784 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T10:32:43.428 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:43.465 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:43.484 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:32:44.355 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T10:32:44.438 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T10:32:44.485 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T10:32:44.596 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T10:32:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T10:32:44.646 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T10:32:44.672 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T10:32:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T10:32:44.699 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:32:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:32:44.807 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T10:32:44.807 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:32:44.892 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:32:44.919 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T10:32:44.960 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T10:32:44.985 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T10:32:45.010 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T10:32:45.045 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T10:32:45.082 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T10:32:45.115 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T10:32:45.140 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T10:32:45.167 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T10:32:45.192 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:32:45.218 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T10:32:45.224 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:45.225 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:45.404 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:32:45.404 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:32:45.470 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:32:45.495 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:32:45.526 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T10:32:45.554 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T10:32:45.580 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T10:32:45.647 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T10:32:50.595 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T10:32:50.596 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T10:32:50.597 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T10:32:50.615 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T10:32:50.615 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T10:32:50.616 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T10:32:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T10:32:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T10:32:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T10:32:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T10:32:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T10:32:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T10:32:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T10:32:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T10:32:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T10:32:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T10:32:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T10:32:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T10:32:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:32:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T10:32:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T10:32:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T10:32:53.475 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:32:53.476 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:32:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.479 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.480 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:32:53.481 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:32:53.482 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:32:53.483 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:32:53.484 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:32:53.485 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:32:53.486 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:32:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:32:53.488 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:32:53.488 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:32:53.488 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:32:53.488 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-24T10:32:53.988 DEBUG:teuthology.parallel:result is None 2024-03-24T10:32:53.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:32:54.137 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:32:54.158 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:32:54.159 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:32:54.159 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:32:54.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:32:54.376 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:32:54.401 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:32:54.402 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:32:54.402 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:32:54.403 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T10:32:54.404 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:54.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:32:54.437 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:54.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:32:54.477 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T10:32:54.477 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:54.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:32:54.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:32:54.576 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:54.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:32:54.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:32:54.678 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T10:32:54.678 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:54.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:32:54.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:32:54.776 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:54.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:32:54.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:32:54.875 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T10:32:54.875 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:54.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:32:54.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:32:54.974 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:54.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:32:55.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:32:55.071 INFO:teuthology.run_tasks:Running task ceph... 2024-03-24T10:32:55.183 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-24T10:32:55.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-03-24T10:32:55.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-03-24T10:32:55.213 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-24T10:32:55.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T10:32:55.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T10:32:55.282 INFO:tasks.ceph:Creating extra log directories... 2024-03-24T10:32:55.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T10:32:55.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T10:32:55.359 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-24T10:32:55.359 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-24T10:32:55.360 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-24T10:32:55.360 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:55.360 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T10:32:55.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T10:32:55.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T10:32:55.551 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:55.551 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T10:32:55.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T10:32:55.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T10:32:55.714 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-24T10:32:55.714 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-24T10:32:55.715 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049', 'branch': 'wip-batrick-testing-20240323.003144-squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7619049', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240323.003144-squid'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240323.003144-squid', 'sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4284, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'sleep_before_teardown': 0, 'subset': '405/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240323.003144-squid', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI9EvjocjrvBu7amPMhnD4sgSivbF05PbumcxFiTQTEAOnMr9s+stLL13DGeLbt4DBy4k42+xo17vHsmObc4vbo=', 'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCdFtkHL/LzKO/bqO1/8DiX80bAYuDCkHNfY0y+tyyywsuYpE5ok+2ZrGqubxi+op9W4OeJOGeX//+Qdh0erhwU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-24_04:56:01', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-24T10:32:55.715 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T10:32:55.718 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T10:32:55.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T10:32:55.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T10:32:55.832 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:55.832 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:32:55.853 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:32:55.853 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-24T10:32:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:32:55.908 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T10:32:55.908 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-24 10:32:45.993652941 +0000 2024-03-24T10:32:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-24 10:30:14.742281556 +0000 2024-03-24T10:32:55.909 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-24 10:30:14.742281556 +0000 2024-03-24T10:32:55.909 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-24 10:30:14.742281556 +0000 2024-03-24T10:32:55.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:32:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-24T10:32:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-24T10:32:55.975 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.8453e-05 s, 5.2 MB/s 2024-03-24T10:32:55.976 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:32:56.032 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:56.092 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-24 10:32:45.993652941 +0000 2024-03-24T10:32:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-24 10:30:15.111277508 +0000 2024-03-24T10:32:56.093 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-24 10:30:15.111277508 +0000 2024-03-24T10:32:56.093 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-24 10:30:15.111277508 +0000 2024-03-24T10:32:56.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:32:56.157 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-24T10:32:56.157 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-24T10:32:56.157 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 7.4328e-05 s, 6.9 MB/s 2024-03-24T10:32:56.158 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:32:56.213 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-24 10:32:45.993652941 +0000 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-24 10:30:15.476273504 +0000 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-24 10:30:15.476273504 +0000 2024-03-24T10:32:56.269 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-24 10:30:15.476273504 +0000 2024-03-24T10:32:56.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:32:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-24T10:32:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-24T10:32:56.334 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.4063e-05 s, 6.1 MB/s 2024-03-24T10:32:56.335 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:32:56.393 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:56.451 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-24 10:32:45.994652930 +0000 2024-03-24T10:32:56.452 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-24 10:30:15.812269818 +0000 2024-03-24T10:32:56.452 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-24 10:30:15.812269818 +0000 2024-03-24T10:32:56.452 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-24 10:30:15.812269818 +0000 2024-03-24T10:32:56.452 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:32:56.516 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-24T10:32:56.516 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-24T10:32:56.516 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 7.9786e-05 s, 6.4 MB/s 2024-03-24T10:32:56.517 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:32:56.575 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:56.575 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:56.575 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:32:56.595 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:32:56.595 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-24 10:32:14.994186827 +0000 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-24 10:30:14.739375534 +0000 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-24 10:30:14.739375534 +0000 2024-03-24T10:32:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-24 10:30:14.739375534 +0000 2024-03-24T10:32:56.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:32:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-24T10:32:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-24T10:32:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000260909 s, 2.0 MB/s 2024-03-24T10:32:56.720 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:32:56.776 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-03-24T10:32:56.832 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-24 10:32:14.995186815 +0000 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-24 10:30:15.108370907 +0000 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-24 10:30:15.108370907 +0000 2024-03-24T10:32:56.833 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-24 10:30:15.108370907 +0000 2024-03-24T10:32:56.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:32:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-24T10:32:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-24T10:32:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000192546 s, 2.7 MB/s 2024-03-24T10:32:56.900 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:32:56.957 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-24 10:32:14.995186815 +0000 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-24 10:30:15.480366242 +0000 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-24 10:30:15.480366242 +0000 2024-03-24T10:32:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-24 10:30:15.480366242 +0000 2024-03-24T10:32:57.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:32:57.087 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-24T10:32:57.087 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-24T10:32:57.087 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000178033 s, 2.9 MB/s 2024-03-24T10:32:57.087 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:32:57.142 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-24 10:32:14.996186802 +0000 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-24 10:30:15.820361978 +0000 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-24 10:30:15.820361978 +0000 2024-03-24T10:32:57.198 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-24 10:30:15.820361978 +0000 2024-03-24T10:32:57.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:32:57.263 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-24T10:32:57.263 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-24T10:32:57.263 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000256807 s, 2.0 MB/s 2024-03-24T10:32:57.265 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:32:57.320 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:57.320 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T10:32:57.321 INFO:tasks.ceph:Generating config... 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] debug client = 20 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] debug ms = 1 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] pid file = 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-24T10:32:57.322 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-24T10:32:57.322 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-24T10:32:57.322 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-24T10:32:57.322 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-24T10:32:57.323 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-24T10:32:57.324 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-24T10:32:57.324 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-24T10:32:57.324 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-24T10:32:57.325 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-24T10:32:57.325 INFO:tasks.ceph:Setting up mon.a... 2024-03-24T10:32:57.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-24T10:32:57.391 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-03-24T10:32:57.393 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-24T10:32:57.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:32:57.468 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.b', '172.21.15.3'), ('mon.c', '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]')] 2024-03-24T10:32:57.468 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-24T10:32:57.469 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:57.469 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T10:32:57.523 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --add b 172.21.15.3 --addv c '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid a0192ab9-51d5-4157-8f81-4a83111a7dfa 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:fsid a0192ab9-51d5-4157-8f81-4a83111a7dfa 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-03-24T10:32:57.609778+0000 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-03-24T10:32:57.609778+0000 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-03-24T10:32:57.609 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-03-24T10:32:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-24T10:32:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.b 2024-03-24T10:32:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-03-24T10:32:57.610 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-24T10:32:57.611 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T10:32:57.629 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a0192ab9-51d5-4157-8f81-4a83111a7dfa... 2024-03-24T10:32:57.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T10:32:57.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T10:32:57.720 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T10:32:57.721 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T10:32:57.722 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790]" 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = a0192ab9-51d5-4157-8f81-4a83111a7dfa 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-24T10:32:57.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.724 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-24T10:32:57.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = bitmap 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-24T10:32:57.726 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-24T10:32:57.727 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T10:32:57.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon down mkfs grace = 300 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon op complaint time = 120 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-24T10:32:57.729 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-24T10:32:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-24T10:32:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T10:32:57.730 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: client mount timeout = 600 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = "" 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout:[mds] 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds = 20 2024-03-24T10:32:57.731 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds balancer = 20 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug frag = True 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug scatterstat = True 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: mds op complaint time = 180 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: mds verify scatter = True 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-24T10:32:57.732 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.733 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-03-24T10:32:57.734 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-03-24T10:32:57.735 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T10:32:57.735 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T10:32:57.736 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-03-24T10:32:57.737 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.1,172.21.15.3,[v2:172.21.15.3:3301,v1:172.21.15.3:6790]" 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = a0192ab9-51d5-4157-8f81-4a83111a7dfa 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-03-24T10:32:57.738 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug trim objects = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.739 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bdev async discard = True 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bdev enable discard = True 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore allocator = bitmap 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 1/20 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 1/20 2024-03-24T10:32:57.740 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-24T10:32:57.741 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-03-24T10:32:57.741 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 4/10 2024-03-24T10:32:57.747 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T10:32:57.747 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-24T10:32:57.748 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-03-24T10:32:57.749 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-03-24T10:32:57.750 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:32:57.750 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.750 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T10:32:57.750 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T10:32:57.750 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.751 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: mon down mkfs grace = 300 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T10:32:57.752 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = "" 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2024-03-24T10:32:57.753 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds balancer = 20 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = True 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 900 2024-03-24T10:32:57.754 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 900 2024-03-24T10:32:57.755 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-24T10:32:57.756 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-24T10:32:57.851 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-24T10:32:57.851 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:57.851 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-24T10:32:57.865 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:57.865 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-24T10:32:57.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:32:57.920 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:57.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T10:32:57.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:32:58.004 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:32:58.004 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:32:58.058 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:32:58.058 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:58.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T10:32:58.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:32:58.102 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:32:58.102 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:32:58.158 INFO:tasks.ceph:Setting up mon nodes... 2024-03-24T10:32:58.159 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-24T10:32:58.159 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T10:32:58.213 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T10:32:58.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T10:32:58.281 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T10:32:58.283 INFO:tasks.ceph:Setting up mds nodes... 2024-03-24T10:32:58.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T10:32:58.338 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T10:32:58.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-24T10:32:58.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T10:32:58.460 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T10:32:58.463 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-24T10:32:58.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T10:32:58.545 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T10:32:58.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-24T10:32:58.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T10:32:58.673 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T10:32:58.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-24T10:32:58.700 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-24T10:32:58.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T10:32:58.742 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-24T10:32:58.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T10:32:58.839 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-24T10:32:58.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T10:32:58.896 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-24T10:32:58.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-24T10:32:58.994 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-24T10:32:59.008 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-24T10:32:59.009 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T10:32:59.009 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-24T10:32:59.040 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:59.040 INFO:tasks.ceph:role: osd.0 2024-03-24T10:32:59.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:32:59.040 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:32:59.107 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:32:59.108 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:32:59.108 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:32:59.108 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:32:59.108 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:32:59.139 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-24T10:32:59.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-24T10:32:59.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-24T10:32:59.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-24T10:32:59.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-24T10:32:59.354 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:59.354 INFO:tasks.ceph:role: osd.1 2024-03-24T10:32:59.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:32:59.354 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:32:59.420 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:32:59.456 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-24T10:32:59.458 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-24T10:32:59.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-24T10:32:59.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-24T10:32:59.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-24T10:32:59.665 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:59.665 INFO:tasks.ceph:role: osd.2 2024-03-24T10:32:59.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:32:59.666 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T10:32:59.738 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:32:59.739 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:32:59.764 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-24T10:32:59.766 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-24T10:32:59.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-24T10:32:59.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-24T10:32:59.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-24T10:32:59.974 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:32:59.975 INFO:tasks.ceph:role: osd.3 2024-03-24T10:32:59.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:32:59.975 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:33:00.041 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:33:00.073 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-24T10:33:00.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-24T10:33:00.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-24T10:33:00.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-24T10:33:00.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.312+0000 7fa435c04640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.312+0000 7fa435c04640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.312+0000 7fa435c04640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.312+0000 7fa435c04640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.312+0000 7fa435c04640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:33:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:00.313+0000 7fa435c04640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-24T10:33:03.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T10:33:03.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.294+0000 7ff114d26640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.295+0000 7ff114d26640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.295+0000 7ff114d26640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.295+0000 7ff114d26640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.295+0000 7ff114d26640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:33:03.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:03.295+0000 7ff114d26640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-24T10:33:06.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T10:33:06.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-24T10:33:06.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-24T10:33:06.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:33:06.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:33:06.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:33:06.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:06.268+0000 7fc9fec19640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-24T10:33:09.114 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T10:33:09.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.237+0000 7fe07ab0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-24T10:33:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.237+0000 7fe07ab0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-24T10:33:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.237+0000 7fe07ab0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:33:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.237+0000 7fe07ab0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:33:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.237+0000 7fe07ab0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:33:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:09.238+0000 7fe07ab0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-24T10:33:12.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T10:33:12.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-24T10:33:12.161 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:33:12.161 INFO:tasks.ceph:role: osd.4 2024-03-24T10:33:12.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:33:12.161 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T10:33:12.239 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:33:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:33:12.740 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-24T10:33:12.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-24T10:33:12.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-24T10:33:12.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-24T10:33:12.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-24T10:33:13.001 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:33:13.001 INFO:tasks.ceph:role: osd.5 2024-03-24T10:33:13.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:33:13.001 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T10:33:13.078 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:33:13.078 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:33:13.078 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:33:13.078 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:33:13.079 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:33:13.518 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-24T10:33:13.522 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-24T10:33:13.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-24T10:33:13.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-24T10:33:13.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-24T10:33:13.723 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:33:13.723 INFO:tasks.ceph:role: osd.6 2024-03-24T10:33:13.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:33:13.723 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:33:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:33:13.800 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:33:14.287 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-24T10:33:14.289 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-24T10:33:14.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-24T10:33:14.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-24T10:33:14.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-24T10:33:14.539 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:33:14.548 INFO:tasks.ceph:role: osd.7 2024-03-24T10:33:14.548 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:33:14.548 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:33:14.615 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:33:15.091 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-24T10:33:15.094 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-24T10:33:15.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-24T10:33:15.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-24T10:33:15.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.356+0000 7fd577e68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-24T10:33:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.356+0000 7fd577e68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-24T10:33:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.356+0000 7fd577e68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:33:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.356+0000 7fd577e68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:33:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.356+0000 7fd577e68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:33:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:15.358+0000 7fd577e68640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-24T10:33:18.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T10:33:18.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.335+0000 7fdb69c8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-24T10:33:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.336+0000 7fdb69c8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-24T10:33:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.336+0000 7fdb69c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:33:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.336+0000 7fdb69c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:33:18.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.336+0000 7fdb69c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:33:18.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:18.336+0000 7fdb69c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-24T10:33:21.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T10:33:21.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:21.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.315+0000 7f52c20ce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-24T10:33:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.315+0000 7f52c20ce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-24T10:33:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.315+0000 7f52c20ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:33:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.315+0000 7f52c20ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:33:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.315+0000 7f52c20ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:33:21.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:21.316+0000 7f52c20ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-24T10:33:24.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T10:33:24.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.290+0000 7fc27d813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-24T10:33:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.291+0000 7fc27d813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-24T10:33:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.291+0000 7fc27d813640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:33:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.291+0000 7fc27d813640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:33:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.291+0000 7fc27d813640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:33:24.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:24.291+0000 7fc27d813640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-24T10:33:25.717 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:33:25.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:33:27.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T10:33:27.176 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-24T10:33:27.176 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-24T10:33:27.208 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-24T10:33:27.278 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-24T10:33:27.348 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-24T10:33:27.416 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.416 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-24T10:33:27.484 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-24T10:33:27.553 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:27.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-24T10:33:27.621 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-24T10:33:27.652 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-24T10:33:27.724 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-24T10:33:27.794 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-24T10:33:27.864 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-24T10:33:27.932 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:27.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-24T10:33:28.001 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:28.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-24T10:33:28.074 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:28.074 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-24T10:33:28.092 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:28.092 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-24T10:33:28.148 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:28.148 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-24T10:33:28.172 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:28.173 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-24T10:33:28.230 INFO:tasks.ceph:Adding keys to all mons... 2024-03-24T10:33:28.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T10:33:28.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmHPHHDBAAIuuLFGLFsRogrQjUx44H7A== 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmJeY4FBAA68H0WQ/mqjsr0lZfBIvuIg== 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmjiKEGxAA9ZjhlLO78jdVtdk4FZILhA== 2024-03-24T10:33:28.273 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-03-24T10:33:28.274 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBcAQBmcF+vEhAA6d6m64/OD24zfbP1RjYR1w== 2024-03-24T10:33:28.274 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-03-24T10:33:28.274 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBfAQBmOi2fERAAJ3x4Y4d96THYK1bBkTqqPg== 2024-03-24T10:33:28.274 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBiAQBmhsQJEBAAGwQdvTTDG3liCqR+8Y1++w== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBlAQBmzW42DhAAopMY+PpT0hSNL1Y/h0rOmQ== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmBLzREBAA9Hph9Xljf0eGaV92TSLlfg== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBm2p+QIBAAokf+P5vhhBqszwKd+fn11w== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBm+dgUKBAAG62jv3f5tbgZc3yHG/BiFQ== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBrAQBmqZxLFRAAoXQjHMDUYKqwpqKunxNuzg== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBuAQBmIQoSFBAAgw2p6P1rqsdn94zL8BCmDg== 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-03-24T10:33:28.275 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBxAQBmnxrVEhAAthwnDMkD5AyCr0gCPvE3ZQ== 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB0AQBm8UNjERAARFrNk44p63MiF0L0YVsAqw== 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout:[client.2] 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBm1Z8YMhAArYu32NrH0OcQYcX13UCsnQ== 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout:[client.1] 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmO0d+NRAAhw9H+eNh7wKil/BAbqbQSQ== 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout:[client.3] 2024-03-24T10:33:28.276 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBm9xNHOxAAAxXBCXt6EiKWgso4flywRQ== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmHPHHDBAAIuuLFGLFsRogrQjUx44H7A== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmJeY4FBAA68H0WQ/mqjsr0lZfBIvuIg== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmjiKEGxAA9ZjhlLO78jdVtdk4FZILhA== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBcAQBmcF+vEhAA6d6m64/OD24zfbP1RjYR1w== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBfAQBmOi2fERAAJ3x4Y4d96THYK1bBkTqqPg== 2024-03-24T10:33:28.297 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-03-24T10:33:28.298 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBiAQBmhsQJEBAAGwQdvTTDG3liCqR+8Y1++w== 2024-03-24T10:33:28.298 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-03-24T10:33:28.300 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBlAQBmzW42DhAAopMY+PpT0hSNL1Y/h0rOmQ== 2024-03-24T10:33:28.300 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-03-24T10:33:28.300 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmBLzREBAA9Hph9Xljf0eGaV92TSLlfg== 2024-03-24T10:33:28.300 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBm2p+QIBAAokf+P5vhhBqszwKd+fn11w== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBm+dgUKBAAG62jv3f5tbgZc3yHG/BiFQ== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBrAQBmqZxLFRAAoXQjHMDUYKqwpqKunxNuzg== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBuAQBmIQoSFBAAgw2p6P1rqsdn94zL8BCmDg== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBxAQBmnxrVEhAAthwnDMkD5AyCr0gCPvE3ZQ== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB0AQBm8UNjERAARFrNk44p63MiF0L0YVsAqw== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout:[client.2] 2024-03-24T10:33:28.301 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBm1Z8YMhAArYu32NrH0OcQYcX13UCsnQ== 2024-03-24T10:33:28.302 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2024-03-24T10:33:28.302 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBmO0d+NRAAhw9H+eNh7wKil/BAbqbQSQ== 2024-03-24T10:33:28.302 INFO:teuthology.orchestra.run.smithi003.stdout:[client.3] 2024-03-24T10:33:28.302 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBaAQBm9xNHOxAAAxXBCXt6EiKWgso4flywRQ== 2024-03-24T10:33:28.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:33:28.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:33:28.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.674 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:33:28.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:33:28.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:33:28.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:28.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:29.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:29.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:29.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:29.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:33:29.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:33:29.346 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-24T10:33:29.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-24T10:33:29.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:33:29.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-24T10:33:29.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-24T10:33:29.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:33:29.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-24T10:33:29.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-24T10:33:29.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:33:29.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-24T10:33:29.834 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:29.837 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:33:29.889 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-24T10:33:29.890 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-24T10:33:29.890 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-24T10:33:29.893 INFO:tasks.ceph.mon.a:Started 2024-03-24T10:33:29.893 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-24T10:33:29.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-24T10:33:29.932 INFO:tasks.ceph.mon.b:Started 2024-03-24T10:33:29.932 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-24T10:33:29.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-24T10:33:29.934 INFO:tasks.ceph.mon.c:Started 2024-03-24T10:33:29.935 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-24T10:33:29.935 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-24T10:33:29.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-24T10:33:29.938 INFO:tasks.ceph.mgr.y:Started 2024-03-24T10:33:29.939 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-24T10:33:29.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-24T10:33:29.943 INFO:tasks.ceph.mgr.x:Started 2024-03-24T10:33:29.943 INFO:tasks.ceph:Setting crush tunables to default 2024-03-24T10:33:29.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-03-24T10:33:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.045+0000 7f909d4d4640 1 Processor -- start 2024-03-24T10:33:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.045+0000 7f909d4d4640 1 -- start start 2024-03-24T10:33:30.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f909d4d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 0x7f9098168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f909d4d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90980b0c40 con 0x7f90980d1c10 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f909d4d4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90980b41e0 con 0x7f909815e460 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f909d4d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f90980cfb90 con 0x7f90980d2ba0 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90977fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 0x7f9098168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90967fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f90980d1c10 0x7f90980d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41774/0 (socket says 172.21.15.1:41774) 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90967fc640 1 -- 172.21.15.1:0/934242655 learned_addr learned my addr 172.21.15.1:0/934242655 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90977fe640 1 -- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 msgr2=0x7f9098168e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90977fe640 1 -- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 msgr2=0x7f9098168e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90977fe640 1 --2- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 0x7f9098168e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:33:30.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.046+0000 7f90977fe640 1 --2- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 0x7f9098168e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:33:30.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846705797 0 0) 0x7f90980b0c40 con 0x7f90980d1c10 2024-03-24T10:33:30.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9080003520 con 0x7f90980d1c10 2024-03-24T10:33:30.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967192856 0 0) 0x7f90980b41e0 con 0x7f909815e460 2024-03-24T10:33:30.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90980b0c40 con 0x7f909815e460 2024-03-24T10:33:30.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607407491 0 0) 0x7f9080003520 con 0x7f90980d1c10 2024-03-24T10:33:30.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 msgr2=0x7f9098168e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 --2- 172.21.15.1:0/934242655 >> v2:172.21.15.1:3300/0 conn(0x7f90980d2ba0 0x7f9098168e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 >> v1:172.21.15.3:6789/0 conn(0x7f909815e460 legacy=0x7f90980cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90980b38b0 con 0x7f90980d1c10 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.221+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f9088001670 con 0x7f90980d1c10 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 >> v1:172.21.15.1:6789/0 conn(0x7f90980d1c10 legacy=0x7f90980d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 0x7f9080003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90800051d0 0x7f908000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f90980cfb90 con 0x7f9080003b50 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f908000c100 con 0x7f90800047c0 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f908000c2c0 con 0x7f90800051d0 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 0x7f9080003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f90967fc640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f90977fe640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90800051d0 0x7f908000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 -- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90800051d0 msgr2=0x7f908000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f90800051d0 0x7f908000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 -- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90800047c0 msgr2=0x7f9080004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f9096ffd640 1 -- 172.21.15.1:0/934242655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9080003520 con 0x7f9080003b50 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.222+0000 7f90977fe640 1 -- 172.21.15.1:0/934242655 reap_dead start 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.223+0000 7f9096ffd640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 0x7f9080003f50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f908c0018a0 tx=0x7f90800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f31964b243fdf0a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.223+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c0052d0 con 0x7f9080003b50 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.223+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c005430 con 0x7f9080003b50 2024-03-24T10:33:30.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.223+0000 7f9095ffb640 1 -- 172.21.15.1:0/934242655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c003810 con 0x7f9080003b50 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 -- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 msgr2=0x7f9080003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 0x7f9080003f50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f908c0018a0 tx=0x7f90800057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 -- 172.21.15.1:0/934242655 shutdown_connections 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 --2- 172.21.15.1:0/934242655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9080003b50 0x7f9080003f50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 -- 172.21.15.1:0/934242655 >> 172.21.15.1:0/934242655 conn(0x7f90980d3eb0 msgr2=0x7f90980d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 -- 172.21.15.1:0/934242655 shutdown_connections 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.224+0000 7f909d4d4640 1 -- 172.21.15.1:0/934242655 wait complete. 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.226+0000 7f909d4d4640 1 Processor -- start 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.226+0000 7f909d4d4640 1 -- start start 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9080003b50 0x7f909815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90981f4ad0 0x7f90981f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f90981f6cf0 con 0x7f90981f38c0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f90981f3e10 con 0x7f90981f4ad0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f909d4d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f90981f40f0 con 0x7f9080003b50 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f90977fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f9096ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9080003b50 0x7f909815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f90977fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55560/0 (socket says 172.21.15.1:55560) 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f90977fe640 1 -- 172.21.15.1:0/1254158270 learned_addr learned my addr 172.21.15.1:0/1254158270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.227+0000 7f90967fc640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90981f4ad0 0x7f90981f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 -- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9080003b50 msgr2=0x7f909815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9080003b50 0x7f909815c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 -- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90981f4ad0 msgr2=0x7f90981f67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90981f4ad0 0x7f90981f67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 -- 172.21.15.1:0/1254158270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90981f43d0 con 0x7f90981f38c0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f90977fe640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f90980ce4b0 tx=0x7f9090007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090009e70 con 0x7f90981f38c0 2024-03-24T10:33:30.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f909000f050 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90981f46b0 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090014650 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90980b80a0 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.228+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9090014b40 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.229+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9098069490 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.230+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f909001d410 con 0x7f90981f38c0 2024-03-24T10:33:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.234+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f909001d5d0 con 0x7f90981f38c0 2024-03-24T10:33:30.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.329+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f90980d2820 con 0x7f90981f38c0 2024-03-24T10:33:30.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.330+0000 7f907f7fe640 1 -- 172.21.15.1:0/1254158270 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f909001d770 con 0x7f90981f38c0 2024-03-24T10:33:30.330 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-24T10:33:30.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 msgr2=0x7f90980c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f90980ce4b0 tx=0x7f9090007700 comp rx=0 tx=0).stop 2024-03-24T10:33:30.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 shutdown_connections 2024-03-24T10:33:30.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90981f38c0 0x7f90980c7ff0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90981f4ad0 0x7f90981f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 --2- 172.21.15.1:0/1254158270 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9080003b50 0x7f909815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 >> 172.21.15.1:0/1254158270 conn(0x7f90980d3eb0 msgr2=0x7f9098158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 shutdown_connections 2024-03-24T10:33:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:30.332+0000 7f909d4d4640 1 -- 172.21.15.1:0/1254158270 wait complete. 2024-03-24T10:33:30.349 INFO:tasks.ceph:check_enable_crimson: False 2024-03-24T10:33:30.349 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-24T10:33:30.349 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:30.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-24T10:33:30.376 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:30.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-24T10:33:30.447 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:30.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-24T10:33:30.518 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:33:30.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-24T10:33:30.589 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:30.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-24T10:33:30.617 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:30.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-24T10:33:30.683 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:30.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-24T10:33:30.704 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:30.703+0000 7f65dd248040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:33:30.730 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:33:30.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-24T10:33:30.732 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:30.733+0000 7f1d1d0fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:33:30.760 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:30.760+0000 7f65dd248040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:33:30.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new b5b76212-d040-46a1-99aa-53cf6f2a06bd 0 2024-03-24T10:33:30.788 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:30.788+0000 7f1d1d0fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:33:30.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.928+0000 7f09f5642640 1 Processor -- start 2024-03-24T10:33:30.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.928+0000 7f09f5642640 1 -- start start 2024-03-24T10:33:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09f5642640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f09f00b4e40 0x7f09f00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09f5642640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f00c7a40 con 0x7f09f0162610 2024-03-24T10:33:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09f5642640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f00b55b0 con 0x7f09f01615f0 2024-03-24T10:33:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09f5642640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f09f00cfd50 con 0x7f09f00b4e40 2024-03-24T10:33:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09ee7fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f09f00b4e40 0x7f09f00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09eeffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f09f01615f0 0x7f09f0161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47512/0 (socket says 172.21.15.3:47512) 2024-03-24T10:33:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09eeffd640 1 -- 172.21.15.3:0/1885046225 learned_addr learned my addr 172.21.15.3:0/1885046225 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09ee7fc640 1 -- 172.21.15.3:0/1885046225 >> v1:172.21.15.3:6789/0 conn(0x7f09f01615f0 legacy=0x7f09f0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09ee7fc640 1 -- 172.21.15.3:0/1885046225 >> v1:172.21.15.1:6789/0 conn(0x7f09f0162610 legacy=0x7f09f00d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.929+0000 7f09ee7fc640 1 -- 172.21.15.3:0/1885046225 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00b52d0 con 0x7f09f00b4e40 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.930+0000 7f09ee7fc640 1 --2- 172.21.15.3:0/1885046225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f00b4e40 0x7f09f00d4a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=601be2050cb98f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.930+0000 7f09edffb640 1 -- 172.21.15.3:0/1885046225 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0030610 con 0x7f09f00b4e40 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.930+0000 7f09edffb640 1 -- 172.21.15.3:0/1885046225 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0030770 con 0x7f09f00b4e40 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.930+0000 7f09edffb640 1 -- 172.21.15.3:0/1885046225 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0030b00 con 0x7f09f00b4e40 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 -- 172.21.15.3:0/1885046225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f00b4e40 msgr2=0x7f09f00d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 --2- 172.21.15.3:0/1885046225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f00b4e40 0x7f09f00d4a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b960 comp rx=0 tx=0).stop 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 -- 172.21.15.3:0/1885046225 shutdown_connections 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 --2- 172.21.15.3:0/1885046225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f00b4e40 0x7f09f00d4a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 -- 172.21.15.3:0/1885046225 >> 172.21.15.3:0/1885046225 conn(0x7f09f00d3ef0 msgr2=0x7f09f00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:30.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 -- 172.21.15.3:0/1885046225 shutdown_connections 2024-03-24T10:33:30.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.931+0000 7f09f5642640 1 -- 172.21.15.3:0/1885046225 wait complete. 2024-03-24T10:33:30.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.932+0000 7f09f5642640 1 Processor -- start 2024-03-24T10:33:30.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.932+0000 7f09f5642640 1 -- start start 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09f01f5450 0x7f09f00cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09f014ac30 con 0x7f09f01f5450 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f09f01f40e0 con 0x7f09f01615f0 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09f5642640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f09f00c8f50 con 0x7f09f0162610 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09eeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:51834/0 (socket says 172.21.15.3:51834) 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09eeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50518/0 (socket says 172.21.15.3:50518) 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 -- 172.21.15.3:0/3468576062 learned_addr learned my addr 172.21.15.3:0/3468576062 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ef7fe640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09f01f5450 0x7f09f00cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 -- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 msgr2=0x7f09f00c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 -- 172.21.15.3:0/3468576062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09f01f5450 msgr2=0x7f09f00cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09f01f5450 0x7f09f00cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 -- 172.21.15.3:0/3468576062 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e0009400 con 0x7f09f0162610 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09eeffd640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.933+0000 7f09ee7fc640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09e002bf70 tx=0x7f09e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e003e040 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0039cb0 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f015f270 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0044630 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f015f830 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f09e0044790 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.934+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f09e004ec20 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.935+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f01f3f20 con 0x7f09f0162610 2024-03-24T10:33:30.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:30.937+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f09f01f3f20 con 0x7f09f0162610 2024-03-24T10:33:31.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.010+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b5b76212-d040-46a1-99aa-53cf6f2a06bd", "id": 0} v 0) -- 0x7f09f00d3040 con 0x7f09f0162610 2024-03-24T10:33:31.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.013+0000 7f09d77fe640 1 -- 172.21.15.3:0/3468576062 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b5b76212-d040-46a1-99aa-53cf6f2a06bd", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f09e0049070 con 0x7f09f0162610 2024-03-24T10:33:31.013 INFO:teuthology.orchestra.run.smithi003.stdout:0 2024-03-24T10:33:31.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 msgr2=0x7f09f015ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09e002bf70 tx=0x7f09e0004730 comp rx=0 tx=0).stop 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 shutdown_connections 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09f01f5450 0x7f09f00cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09f0162610 0x7f09f015ebe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 --2- 172.21.15.3:0/3468576062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09f01615f0 0x7f09f00c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 >> 172.21.15.3:0/3468576062 conn(0x7f09f00d3ef0 msgr2=0x7f09f015cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 shutdown_connections 2024-03-24T10:33:31.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.014+0000 7f09f5642640 1 -- 172.21.15.3:0/3468576062 wait complete. 2024-03-24T10:33:31.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new beba3353-6c19-4121-adea-b93d378c084a 1 2024-03-24T10:33:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.125+0000 7f4be80e8640 1 Processor -- start 2024-03-24T10:33:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 -- start start 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4be0161870 0x7f4be006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4be00d09a0 0x7f4be006ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be0161c50 con 0x7f4be006cf50 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4be006d3e0 con 0x7f4be0161870 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.126+0000 7f4be80e8640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4be00cc350 con 0x7f4be00d09a0 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4be00d09a0 0x7f4be006ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be6685640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4be006cf50 0x7f4be00ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47520/0 (socket says 172.21.15.3:47520) 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4be00d09a0 0x7f4be006ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50520/0 (socket says 172.21.15.3:50520) 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be6685640 1 -- 172.21.15.3:0/2437648866 learned_addr learned my addr 172.21.15.3:0/2437648866 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5e84640 1 --2- 172.21.15.3:0/2437648866 >> v2:172.21.15.1:3300/0 conn(0x7f4be0161870 0x7f4be006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 -- 172.21.15.3:0/2437648866 >> v2:172.21.15.1:3300/0 conn(0x7f4be0161870 msgr2=0x7f4be006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 --2- 172.21.15.3:0/2437648866 >> v2:172.21.15.1:3300/0 conn(0x7f4be0161870 0x7f4be006c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 -- 172.21.15.3:0/2437648866 >> v1:172.21.15.3:6789/0 conn(0x7f4be006cf50 legacy=0x7f4be00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be4e82640 1 -- 172.21.15.3:0/2437648866 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964154737 0 0) 0x7f4be0161c50 con 0x7f4be006cf50 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.127+0000 7f4be5683640 1 -- 172.21.15.3:0/2437648866 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be00b1380 con 0x7f4be00d09a0 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be5683640 1 --2- 172.21.15.3:0/2437648866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be006ca10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4be006ed30 tx=0x7f4bd002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96fa6cc2ca0d9fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be4e82640 1 -- 172.21.15.3:0/2437648866 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd002e070 con 0x7f4be00d09a0 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be4e82640 1 -- 172.21.15.3:0/2437648866 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd00315a0 con 0x7f4be00d09a0 2024-03-24T10:33:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be4e82640 1 -- 172.21.15.3:0/2437648866 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0031930 con 0x7f4be00d09a0 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 -- 172.21.15.3:0/2437648866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 msgr2=0x7f4be006ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 --2- 172.21.15.3:0/2437648866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be006ca10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4be006ed30 tx=0x7f4bd002b840 comp rx=0 tx=0).stop 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 -- 172.21.15.3:0/2437648866 shutdown_connections 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 --2- 172.21.15.3:0/2437648866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be006ca10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 --2- 172.21.15.3:0/2437648866 >> v2:172.21.15.1:3300/0 conn(0x7f4be0161870 0x7f4be006c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.128+0000 7f4be80e8640 1 -- 172.21.15.3:0/2437648866 >> 172.21.15.3:0/2437648866 conn(0x7f4be00d1d00 msgr2=0x7f4be00d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.129+0000 7f4be80e8640 1 -- 172.21.15.3:0/2437648866 shutdown_connections 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.129+0000 7f4be80e8640 1 -- 172.21.15.3:0/2437648866 wait complete. 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 Processor -- start 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 -- start start 2024-03-24T10:33:31.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 0x7f4be01544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 0x7f4be015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0154a20 con 0x7f4be0161870 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0156b70 con 0x7f4be00d09a0 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be80e8640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4be0156e50 con 0x7f4be006cf50 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be5683640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be5e84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 0x7f4be01544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be5683640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50532/0 (socket says 172.21.15.3:50532) 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.130+0000 7f4be5683640 1 -- 172.21.15.3:0/3275924922 learned_addr learned my addr 172.21.15.3:0/3275924922 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:31.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be6685640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 0x7f4be015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 -- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 msgr2=0x7f4be01544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 0x7f4be01544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 -- 172.21.15.3:0/3275924922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 msgr2=0x7f4be015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 0x7f4be015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 -- 172.21.15.3:0/3275924922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd0009400 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5e84640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 0x7f4be01544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be6685640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 0x7f4be015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be5683640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009890 tx=0x7f4bd00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0032840 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd0031f00 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be0157130 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0032be0 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be0156680 con 0x7f4be00d09a0 2024-03-24T10:33:31.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4bd0005d20 con 0x7f4be00d09a0 2024-03-24T10:33:31.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.131+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f4bd0045880 con 0x7f4be00d09a0 2024-03-24T10:33:31.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.132+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be004f470 con 0x7f4be00d09a0 2024-03-24T10:33:31.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.134+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4bd0063dd0 con 0x7f4be00d09a0 2024-03-24T10:33:31.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.208+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "beba3353-6c19-4121-adea-b93d378c084a", "id": 1} v 0) -- 0x7f4be015f490 con 0x7f4be00d09a0 2024-03-24T10:33:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.212+0000 7f4bd67fc640 1 -- 172.21.15.3:0/3275924922 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "beba3353-6c19-4121-adea-b93d378c084a", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f4bd00520b0 con 0x7f4be00d09a0 2024-03-24T10:33:31.212 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 msgr2=0x7f4be0156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009890 tx=0x7f4bd00046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 shutdown_connections 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be0161870 0x7f4be015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be00d09a0 0x7f4be0156140 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 --2- 172.21.15.3:0/3275924922 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be006cf50 0x7f4be01544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 >> 172.21.15.3:0/3275924922 conn(0x7f4be00d1d00 msgr2=0x7f4be006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 shutdown_connections 2024-03-24T10:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.213+0000 7f4be80e8640 1 -- 172.21.15.3:0/3275924922 wait complete. 2024-03-24T10:33:31.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 7c577a7b-20a1-4907-b379-3674f2cb3caf 2 2024-03-24T10:33:31.313 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.312+0000 7f65dd248040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:33:31.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 Processor -- start 2024-03-24T10:33:31.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 -- start start 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9a01615f0 0x7fb9a00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00c7a40 con 0x7fb9a00b4e40 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00b55b0 con 0x7fb9a0162610 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.325+0000 7fb9a58fa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a00cfd50 con 0x7fb9a01615f0 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99f7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb9a0162610 0x7fb9a00d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47528/0 (socket says 172.21.15.3:47528) 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99f7fe640 1 -- 172.21.15.3:0/2457505456 learned_addr learned my addr 172.21.15.3:0/2457505456 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99effd640 1 --2- 172.21.15.3:0/2457505456 >> v2:172.21.15.1:3300/0 conn(0x7fb9a01615f0 0x7fb9a00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496726611 0 0) 0x7fb9a00b55b0 con 0x7fb9a0162610 2024-03-24T10:33:31.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb988003520 con 0x7fb9a0162610 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294716498 0 0) 0x7fb9a00c7a40 con 0x7fb9a00b4e40 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a00b55b0 con 0x7fb9a00b4e40 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99effd640 1 -- 172.21.15.3:0/2457505456 >> v1:172.21.15.3:6789/0 conn(0x7fb9a0162610 legacy=0x7fb9a00d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99effd640 1 -- 172.21.15.3:0/2457505456 >> v1:172.21.15.1:6789/0 conn(0x7fb9a00b4e40 legacy=0x7fb9a0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.326+0000 7fb99effd640 1 -- 172.21.15.3:0/2457505456 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a00b52d0 con 0x7fb9a01615f0 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb99effd640 1 --2- 172.21.15.3:0/2457505456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a01615f0 0x7fb9a00d4a10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb9900097a0 tx=0x7fb99002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89e1822a4106237 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990030610 con 0x7fb9a01615f0 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb990030770 con 0x7fb9a01615f0 2024-03-24T10:33:31.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb99dffb640 1 -- 172.21.15.3:0/2457505456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990030b00 con 0x7fb9a01615f0 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb9a58fa640 1 -- 172.21.15.3:0/2457505456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a01615f0 msgr2=0x7fb9a00d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.327+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/2457505456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a01615f0 0x7fb9a00d4a10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb9900097a0 tx=0x7fb99002b960 comp rx=0 tx=0).stop 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.328+0000 7fb9a58fa640 1 -- 172.21.15.3:0/2457505456 shutdown_connections 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.328+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/2457505456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a01615f0 0x7fb9a00d4a10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.328+0000 7fb9a58fa640 1 -- 172.21.15.3:0/2457505456 >> 172.21.15.3:0/2457505456 conn(0x7fb9a00d3ef0 msgr2=0x7fb9a00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.328+0000 7fb9a58fa640 1 -- 172.21.15.3:0/2457505456 shutdown_connections 2024-03-24T10:33:31.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.328+0000 7fb9a58fa640 1 -- 172.21.15.3:0/2457505456 wait complete. 2024-03-24T10:33:31.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.329+0000 7fb9a58fa640 1 Processor -- start 2024-03-24T10:33:31.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.329+0000 7fb9a58fa640 1 -- start start 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9a01615f0 0x7fb9a014a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 0x7fb9a00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a00c9970 con 0x7fb9a0162610 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a00cd950 con 0x7fb9a0158f00 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a00cd390 con 0x7fb9a01615f0 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99effd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9a01615f0 0x7fb9a014a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50544/0 (socket says 172.21.15.3:50544) 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 -- 172.21.15.3:0/1213915627 learned_addr learned my addr 172.21.15.3:0/1213915627 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99e7fc640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 0x7fb9a00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 -- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9a01615f0 msgr2=0x7fb9a014a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9a01615f0 0x7fb9a014a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 -- 172.21.15.3:0/1213915627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 msgr2=0x7fb9a00cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 0x7fb9a00cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 -- 172.21.15.3:0/1213915627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb990009400 con 0x7fb9a0158f00 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99e7fc640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 0x7fb9a00cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb99f7fe640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb99800caa0 tx=0x7fb99800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99800ac00 con 0x7fb9a0158f00 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99800ad80 con 0x7fb9a0158f00 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998012640 con 0x7fb9a0158f00 2024-03-24T10:33:31.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.330+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a00cd670 con 0x7fb9a0158f00 2024-03-24T10:33:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.331+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a01482b0 con 0x7fb9a0158f00 2024-03-24T10:33:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.331+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb9980127a0 con 0x7fb9a0158f00 2024-03-24T10:33:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.331+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fb998007490 con 0x7fb9a0158f00 2024-03-24T10:33:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.332+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a01484d0 con 0x7fb9a0158f00 2024-03-24T10:33:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.334+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb998017070 con 0x7fb9a0158f00 2024-03-24T10:33:31.336 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.336+0000 7f1d1d0fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:33:31.370 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.370+0000 7f65dd248040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:33:31.398 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.392+0000 7f1d1d0fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:33:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.408+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7c577a7b-20a1-4907-b379-3674f2cb3caf", "id": 2} v 0) -- 0x7fb9a004f470 con 0x7fb9a0158f00 2024-03-24T10:33:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.411+0000 7fb987fff640 1 -- 172.21.15.3:0/1213915627 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c577a7b-20a1-4907-b379-3674f2cb3caf", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fb998012d50 con 0x7fb9a0158f00 2024-03-24T10:33:31.411 INFO:teuthology.orchestra.run.smithi003.stdout:2 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 msgr2=0x7fb9a0155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb99800caa0 tx=0x7fb99800cf60 comp rx=0 tx=0).stop 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 shutdown_connections 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9a0158f00 0x7fb9a0155ef0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a0162610 0x7fb9a00cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 --2- 172.21.15.3:0/1213915627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9a01615f0 0x7fb9a014a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 >> 172.21.15.3:0/1213915627 conn(0x7fb9a00d3ef0 msgr2=0x7fb9a006c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 shutdown_connections 2024-03-24T10:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.412+0000 7fb9a58fa640 1 -- 172.21.15.3:0/1213915627 wait complete. 2024-03-24T10:33:31.420 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.418+0000 7f65dd248040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:33:31.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 89097d54-de31-43a5-b6ff-3ad809e70b6f 3 2024-03-24T10:33:31.439 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.439+0000 7f1d1d0fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:33:31.475 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.475+0000 7f65dd248040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:33:31.494 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.494+0000 7f1d1d0fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:33:31.520 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.520+0000 7f65dd248040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:33:31.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.532+0000 7f6c479a1640 1 Processor -- start 2024-03-24T10:33:31.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.532+0000 7f6c479a1640 1 -- start start 2024-03-24T10:33:31.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c479a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c400d06a0 0x7f6c400cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c479a1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4015c590 con 0x7f6c400d28b0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c479a1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c400cdb50 con 0x7f6c4015c1b0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c479a1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6c400cd2b0 con 0x7f6c400d06a0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c4573d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6c4015c1b0 0x7f6c400cff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47542/0 (socket says 172.21.15.3:47542) 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c4573d640 1 -- 172.21.15.3:0/2605043123 learned_addr learned my addr 172.21.15.3:0/2605043123 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c44f3c640 1 --2- 172.21.15.3:0/2605043123 >> v2:172.21.15.1:3300/0 conn(0x7f6c400d06a0 0x7f6c400cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c37fff640 1 -- 172.21.15.3:0/2605043123 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054547006 0 0) 0x7f6c400cdb50 con 0x7f6c4015c1b0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c37fff640 1 -- 172.21.15.3:0/2605043123 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c2c003520 con 0x7f6c4015c1b0 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c44f3c640 1 -- 172.21.15.3:0/2605043123 >> v1:172.21.15.3:6789/0 conn(0x7f6c4015c1b0 legacy=0x7f6c400cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c44f3c640 1 -- 172.21.15.3:0/2605043123 >> v1:172.21.15.1:6789/0 conn(0x7f6c400d28b0 legacy=0x7f6c400c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.533+0000 7f6c44f3c640 1 -- 172.21.15.3:0/2605043123 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c400cd590 con 0x7f6c400d06a0 2024-03-24T10:33:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.534+0000 7f6c44f3c640 1 --2- 172.21.15.3:0/2605043123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c400d06a0 0x7f6c400cf8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c300097a0 tx=0x7f6c3002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbac602c950df3c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.534+0000 7f6c37fff640 1 -- 172.21.15.3:0/2605043123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30030610 con 0x7f6c400d06a0 2024-03-24T10:33:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.534+0000 7f6c37fff640 1 -- 172.21.15.3:0/2605043123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c30030770 con 0x7f6c400d06a0 2024-03-24T10:33:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.534+0000 7f6c37fff640 1 -- 172.21.15.3:0/2605043123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30030ae0 con 0x7f6c400d06a0 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 -- 172.21.15.3:0/2605043123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c400d06a0 msgr2=0x7f6c400cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2605043123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c400d06a0 0x7f6c400cf8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c300097a0 tx=0x7f6c3002b960 comp rx=0 tx=0).stop 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 -- 172.21.15.3:0/2605043123 shutdown_connections 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2605043123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c400d06a0 0x7f6c400cf8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 -- 172.21.15.3:0/2605043123 >> 172.21.15.3:0/2605043123 conn(0x7f6c400d1d00 msgr2=0x7f6c400d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 -- 172.21.15.3:0/2605043123 shutdown_connections 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.535+0000 7f6c479a1640 1 -- 172.21.15.3:0/2605043123 wait complete. 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.536+0000 7f6c479a1640 1 Processor -- start 2024-03-24T10:33:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.536+0000 7f6c479a1640 1 -- start start 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c400d28b0 0x7f6c4015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 0x7f6c4006cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4015e1d0 con 0x7f6c4015c1b0 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4015efc0 con 0x7f6c400d06a0 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c479a1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6c4006d100 con 0x7f6c400d28b0 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50552/0 (socket says 172.21.15.3:50552) 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 -- 172.21.15.3:0/2141327497 learned_addr learned my addr 172.21.15.3:0/2141327497 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c44f3c640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c400d28b0 0x7f6c4015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c45f3e640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 0x7f6c4006cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 -- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c400d28b0 msgr2=0x7f6c4015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c400d28b0 0x7f6c4015dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 -- 172.21.15.3:0/2141327497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 msgr2=0x7f6c4006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 0x7f6c4006cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c4573d640 1 -- 172.21.15.3:0/2141327497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c30009400 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.537+0000 7f6c45f3e640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 0x7f6c4006cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c4573d640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c410 tx=0x7f6c3c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c005ce0 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3c005e40 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c0072f0 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4015e3b0 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6c3c007450 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c4015e970 con 0x7f6c400d06a0 2024-03-24T10:33:31.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.538+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f6c3c012890 con 0x7f6c400d06a0 2024-03-24T10:33:31.541 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.538+0000 7f1d1d0fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:33:31.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.539+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4004f470 con 0x7f6c400d06a0 2024-03-24T10:33:31.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.541+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6c3c017070 con 0x7f6c400d06a0 2024-03-24T10:33:31.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.630+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "89097d54-de31-43a5-b6ff-3ad809e70b6f", "id": 3} v 0) -- 0x7f6c400cf590 con 0x7f6c400d06a0 2024-03-24T10:33:31.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.633+0000 7f6c35ffb640 1 -- 172.21.15.3:0/2141327497 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89097d54-de31-43a5-b6ff-3ad809e70b6f", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f6c3c012ba0 con 0x7f6c400d06a0 2024-03-24T10:33:31.633 INFO:teuthology.orchestra.run.smithi003.stdout:3 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 msgr2=0x7f6c4015f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c410 tx=0x7f6c3c00c8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 shutdown_connections 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4015c1b0 0x7f6c4006cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c400d28b0 0x7f6c4015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 --2- 172.21.15.3:0/2141327497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c400d06a0 0x7f6c4015f450 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 >> 172.21.15.3:0/2141327497 conn(0x7f6c400d1d00 msgr2=0x7f6c400cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 shutdown_connections 2024-03-24T10:33:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.634+0000 7f6c479a1640 1 -- 172.21.15.3:0/2141327497 wait complete. 2024-03-24T10:33:31.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 1b743f53-f298-412e-9fe7-69695fbe74a8 4 2024-03-24T10:33:31.655 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.655+0000 7f65dd248040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:33:31.658 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.658+0000 7f1d1d0fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:33:31.731 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.730+0000 7f1d1d0fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:33:31.731 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.730+0000 7f65dd248040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:33:31.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 Processor -- start 2024-03-24T10:33:31.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 -- start start 2024-03-24T10:33:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 0x7fea500d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 0x7fea50161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea500c7a40 con 0x7fea501619d0 2024-03-24T10:33:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fea500cfd50 con 0x7fea501615f0 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.745+0000 7fea550bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fea500b55b0 con 0x7fea500b4e40 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4e59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 0x7fea50161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4f59e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fea501619d0 0x7fea500d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47548/0 (socket says 172.21.15.3:47548) 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4f59e640 1 -- 172.21.15.3:0/3425724381 learned_addr learned my addr 172.21.15.3:0/3425724381 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4ed9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 0x7fea500d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4e59c640 1 -- 172.21.15.3:0/3425724381 >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 msgr2=0x7fea500d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4e59c640 1 --2- 172.21.15.3:0/3425724381 >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 0x7fea500d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4e59c640 1 -- 172.21.15.3:0/3425724381 >> v1:172.21.15.3:6789/0 conn(0x7fea501619d0 legacy=0x7fea500d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4e59c640 1 -- 172.21.15.3:0/3425724381 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea500b52d0 con 0x7fea500b4e40 2024-03-24T10:33:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.746+0000 7fea4ed9d640 1 --2- 172.21.15.3:0/3425724381 >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 0x7fea500d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.747+0000 7fea4e59c640 1 --2- 172.21.15.3:0/3425724381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 0x7fea50161f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea440097a0 tx=0x7fea4402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d0206266bbde739 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.747+0000 7fea4dd9b640 1 -- 172.21.15.3:0/3425724381 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44031830 con 0x7fea500b4e40 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.747+0000 7fea4dd9b640 1 -- 172.21.15.3:0/3425724381 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea44031990 con 0x7fea500b4e40 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.747+0000 7fea4dd9b640 1 -- 172.21.15.3:0/3425724381 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44031d20 con 0x7fea500b4e40 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 -- 172.21.15.3:0/3425724381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 msgr2=0x7fea50161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 --2- 172.21.15.3:0/3425724381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 0x7fea50161f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea440097a0 tx=0x7fea4402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 -- 172.21.15.3:0/3425724381 shutdown_connections 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 --2- 172.21.15.3:0/3425724381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b4e40 0x7fea50161f00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 --2- 172.21.15.3:0/3425724381 >> v2:172.21.15.3:3300/0 conn(0x7fea501615f0 0x7fea500d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 -- 172.21.15.3:0/3425724381 >> 172.21.15.3:0/3425724381 conn(0x7fea500d3ef0 msgr2=0x7fea500d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 -- 172.21.15.3:0/3425724381 shutdown_connections 2024-03-24T10:33:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.748+0000 7fea550bb640 1 -- 172.21.15.3:0/3425724381 wait complete. 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.749+0000 7fea550bb640 1 Processor -- start 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.749+0000 7fea550bb640 1 -- start start 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 0x7fea501497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 0x7fea500c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fea500b7470 con 0x7fea501615f0 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fea501f82d0 con 0x7fea501619d0 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea550bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fea50149d30 con 0x7fea500b6a80 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea4f59e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea4e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 0x7fea500c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea4f59e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:51874/0 (socket says 172.21.15.3:51874) 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea4f59e640 1 -- 172.21.15.3:0/3737574682 learned_addr learned my addr 172.21.15.3:0/3737574682 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.750+0000 7fea4ed9d640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 0x7fea501497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 -- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 msgr2=0x7fea500b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 -- 172.21.15.3:0/3737574682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 msgr2=0x7fea501497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 0x7fea501497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 -- 172.21.15.3:0/3737574682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea40009400 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4ed9d640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 0x7fea501497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4f59e640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea4e59c640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 0x7fea500c7140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea500b56f0 tx=0x7fea440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4403a6f0 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea4403a870 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea44009400 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4403ac20 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea501f7a30 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea4404d440 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.751+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fea440439c0 con 0x7fea501619d0 2024-03-24T10:33:31.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.752+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea5004f470 con 0x7fea501619d0 2024-03-24T10:33:31.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.754+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fea44048070 con 0x7fea501619d0 2024-03-24T10:33:31.778 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.778+0000 7f1d1d0fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:33:31.780 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.780+0000 7f65dd248040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:33:31.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.828+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1b743f53-f298-412e-9fe7-69695fbe74a8", "id": 4} v 0) -- 0x7fea5008d640 con 0x7fea501619d0 2024-03-24T10:33:31.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.831+0000 7fea377fe640 1 -- 172.21.15.3:0/3737574682 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1b743f53-f298-412e-9fe7-69695fbe74a8", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fea44035020 con 0x7fea501619d0 2024-03-24T10:33:31.831 INFO:teuthology.orchestra.run.smithi003.stdout:4 2024-03-24T10:33:31.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 msgr2=0x7fea500c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 0x7fea500c7140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fea500b56f0 tx=0x7fea440046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 shutdown_connections 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea500b6a80 0x7fea500b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501619d0 0x7fea500c7140 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 --2- 172.21.15.3:0/3737574682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea501615f0 0x7fea501497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 >> 172.21.15.3:0/3737574682 conn(0x7fea500d3ef0 msgr2=0x7fea500d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 shutdown_connections 2024-03-24T10:33:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.832+0000 7fea550bb640 1 -- 172.21.15.3:0/3737574682 wait complete. 2024-03-24T10:33:31.833 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.832+0000 7f1d1d0fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:33:31.837 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.835+0000 7f65dd248040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:33:31.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new aec7a526-7963-4327-9f94-9bcc1bc8c669 5 2024-03-24T10:33:31.881 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:31.881+0000 7f1d1d0fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:33:31.884 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:31.884+0000 7f65dd248040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:33:31.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.984+0000 7f29a6855640 1 Processor -- start 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.984+0000 7f29a6855640 1 -- start start 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f29a6855640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29a0162610 0x7f29a00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f29a6855640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a00c7a40 con 0x7f29a01615f0 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f29a6855640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a00b55b0 con 0x7f29a00b4e40 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f29a6855640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f29a00cfd50 con 0x7f29a0162610 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f299f7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f29a00b4e40 0x7f29a00d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47564/0 (socket says 172.21.15.3:47564) 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f299f7fe640 1 -- 172.21.15.3:0/2301907638 learned_addr learned my addr 172.21.15.3:0/2301907638 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2301907638 >> v2:172.21.15.1:3300/0 conn(0x7f29a0162610 0x7f29a00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.985+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195002164 0 0) 0x7f29a00b55b0 con 0x7f29a00b4e40 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2988003520 con 0x7f29a00b4e40 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067800558 0 0) 0x7f29a00c7a40 con 0x7f29a01615f0 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a00b55b0 con 0x7f29a01615f0 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4161353206 0 0) 0x7f2988003520 con 0x7f29a00b4e40 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 >> v2:172.21.15.1:3300/0 conn(0x7f29a0162610 msgr2=0x7f29a00d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:31.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 --2- 172.21.15.3:0/2301907638 >> v2:172.21.15.1:3300/0 conn(0x7f29a0162610 0x7f29a00d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 >> v1:172.21.15.1:6789/0 conn(0x7f29a01615f0 legacy=0x7f29a0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a00b52d0 con 0x7f29a00b4e40 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f2990001670 con 0x7f29a00b4e40 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 >> v1:172.21.15.3:6789/0 conn(0x7f29a00b4e40 legacy=0x7f29a00d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2988003b00 0x7f2988003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 0x7f2988004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2988005180 0x7f298800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29a00cfd50 con 0x7f2988005180 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f298800c000 con 0x7f29880047c0 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f298800c1c0 con 0x7f2988003b00 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2301907638 >> v2:172.21.15.1:3300/0 conn(0x7f29a0162610 0x7f29a00d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 0x7f2988004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299ffff640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2988003b00 0x7f2988003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2988005180 0x7f298800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 -- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2988003b00 msgr2=0x7f2988003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:31.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2988003b00 0x7f2988003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 -- 172.21.15.3:0/2301907638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2988005180 msgr2=0x7f298800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2988005180 0x7f298800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f299f7fe640 1 -- 172.21.15.3:0/2301907638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2988003520 con 0x7f29880047c0 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.986+0000 7f29a4df2640 1 -- 172.21.15.3:0/2301907638 reap_dead start 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.987+0000 7f299f7fe640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 0x7f2988004c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29a008c3c0 tx=0x7f29880057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55db654c166a8bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.987+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990005ad0 con 0x7f29880047c0 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.987+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990005c30 con 0x7f29880047c0 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.987+0000 7f299effd640 1 -- 172.21.15.3:0/2301907638 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299002c7c0 con 0x7f29880047c0 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 -- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 msgr2=0x7f2988004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 0x7f2988004c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29a008c3c0 tx=0x7f29880057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 -- 172.21.15.3:0/2301907638 shutdown_connections 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 --2- 172.21.15.3:0/2301907638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29880047c0 0x7f2988004c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 -- 172.21.15.3:0/2301907638 >> 172.21.15.3:0/2301907638 conn(0x7f29a00d3ef0 msgr2=0x7f29a00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 -- 172.21.15.3:0/2301907638 shutdown_connections 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.988+0000 7f29a6855640 1 -- 172.21.15.3:0/2301907638 wait complete. 2024-03-24T10:33:31.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.989+0000 7f29a6855640 1 Processor -- start 2024-03-24T10:33:31.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.989+0000 7f29a6855640 1 -- start start 2024-03-24T10:33:31.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 0x7f29a0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:31.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29a01584a0 con 0x7f29880047c0 2024-03-24T10:33:31.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0158a20 con 0x7f29a00cc550 2024-03-24T10:33:31.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a6855640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f29a00b6e20 con 0x7f29a0155f10 2024-03-24T10:33:32.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299f7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299f7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.3:51896/0 (socket says 172.21.15.3:51896) 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50574/0 (socket says 172.21.15.3:50574) 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299f7fe640 1 -- 172.21.15.3:0/2592832985 learned_addr learned my addr 172.21.15.3:0/2592832985 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299ffff640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 0x7f29a0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 -- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 msgr2=0x7f29a00cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 -- 172.21.15.3:0/2592832985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 msgr2=0x7f29a0158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 0x7f29a0158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 -- 172.21.15.3:0/2592832985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a00b7100 con 0x7f29a00cc550 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299f7fe640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f299ffff640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 0x7f29a0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.990+0000 7f29a4df2640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f29a00d4f80 tx=0x7f299800f580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299800e040 con 0x7f29a00cc550 2024-03-24T10:33:32.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f299800f040 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998003bd0 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a00b73e0 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a01557c0 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f29980157c0 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.991+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f299801da90 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.992+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a004f470 con 0x7f29a00cc550 2024-03-24T10:33:32.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:31.994+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2998022060 con 0x7f29a00cc550 2024-03-24T10:33:32.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.068+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "aec7a526-7963-4327-9f94-9bcc1bc8c669", "id": 5} v 0) -- 0x7f29a0158b60 con 0x7f29a00cc550 2024-03-24T10:33:32.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.071+0000 7f299cff9640 1 -- 172.21.15.3:0/2592832985 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aec7a526-7963-4327-9f94-9bcc1bc8c669", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f299801d4d0 con 0x7f29a00cc550 2024-03-24T10:33:32.071 INFO:teuthology.orchestra.run.smithi003.stdout:5 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.071+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 msgr2=0x7f29a00cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.071+0000 7f29a6855640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f29a00d4f80 tx=0x7f299800f580 comp rx=0 tx=0).stop 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 shutdown_connections 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29a00cc550 0x7f29a00cc9d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29a0155f10 0x7f29a00cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 --2- 172.21.15.3:0/2592832985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29880047c0 0x7f29a0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 >> 172.21.15.3:0/2592832985 conn(0x7f29a00d3ef0 msgr2=0x7f29a00c98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 shutdown_connections 2024-03-24T10:33:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.072+0000 7f29a6855640 1 -- 172.21.15.3:0/2592832985 wait complete. 2024-03-24T10:33:32.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new b38bb608-40ac-4f3f-9ece-e4d002045f3d 6 2024-03-24T10:33:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.183+0000 7f3ecb75c640 1 Processor -- start 2024-03-24T10:33:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.183+0000 7f3ecb75c640 1 -- start start 2024-03-24T10:33:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ecb75c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ec406b7b0 0x7f3ec40cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ecb75c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ec40b3310 0x7f3ec4072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ecb75c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec4072620 con 0x7f3ec415e320 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ecb75c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec40d02b0 con 0x7f3ec406b7b0 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ecb75c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec415e700 con 0x7f3ec40b3310 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ec40b3310 0x7f3ec4072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ec40b3310 0x7f3ec4072ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50584/0 (socket says 172.21.15.3:50584) 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec94f8640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3ec415e320 0x7f3ec406b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47574/0 (socket says 172.21.15.3:47574) 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 -- 172.21.15.3:0/3526425838 learned_addr learned my addr 172.21.15.3:0/3526425838 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec8cf7640 1 --2- 172.21.15.3:0/3526425838 >> v2:172.21.15.1:3300/0 conn(0x7f3ec406b7b0 0x7f3ec40cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 -- 172.21.15.3:0/3526425838 >> v2:172.21.15.1:3300/0 conn(0x7f3ec406b7b0 msgr2=0x7f3ec40cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 --2- 172.21.15.3:0/3526425838 >> v2:172.21.15.1:3300/0 conn(0x7f3ec406b7b0 0x7f3ec40cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.184+0000 7f3ec9cf9640 1 -- 172.21.15.3:0/3526425838 >> v1:172.21.15.3:6789/0 conn(0x7f3ec415e320 legacy=0x7f3ec406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3ec9cf9640 1 -- 172.21.15.3:0/3526425838 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec40cfcf0 con 0x7f3ec40b3310 2024-03-24T10:33:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3eb7fff640 1 -- 172.21.15.3:0/3526425838 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3865608914 0 0) 0x7f3ec4072620 con 0x7f3ec415e320 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3ec9cf9640 1 --2- 172.21.15.3:0/3526425838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4072ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40b5fc0 tx=0x7f3ebc02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4a8bb0c107ffbf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3eb7fff640 1 -- 172.21.15.3:0/3526425838 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc02ca80 con 0x7f3ec40b3310 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3eb7fff640 1 -- 172.21.15.3:0/3526425838 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ebc02cbe0 con 0x7f3ec40b3310 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.185+0000 7f3eb7fff640 1 -- 172.21.15.3:0/3526425838 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc0343f0 con 0x7f3ec40b3310 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 -- 172.21.15.3:0/3526425838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 msgr2=0x7f3ec4072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/3526425838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4072ab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40b5fc0 tx=0x7f3ebc02bee0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 -- 172.21.15.3:0/3526425838 shutdown_connections 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/3526425838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4072ab0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/3526425838 >> v2:172.21.15.1:3300/0 conn(0x7f3ec406b7b0 0x7f3ec40cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 -- 172.21.15.3:0/3526425838 >> 172.21.15.3:0/3526425838 conn(0x7f3ec40b2820 msgr2=0x7f3ec40b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 -- 172.21.15.3:0/3526425838 shutdown_connections 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.186+0000 7f3ecb75c640 1 -- 172.21.15.3:0/3526425838 wait complete. 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.187+0000 7f3ecb75c640 1 Processor -- start 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.187+0000 7f3ecb75c640 1 -- start start 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec406b7b0 0x7f3ec40b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 0x7f3ec415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec41f4ba0 con 0x7f3ec406b7b0 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec40b5ce0 con 0x7f3ec40b3310 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ecb75c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec41f3cf0 con 0x7f3ec415e320 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec9cf9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 0x7f3ec415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50596/0 (socket says 172.21.15.3:50596) 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 -- 172.21.15.3:0/2640637792 learned_addr learned my addr 172.21.15.3:0/2640637792 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec94f8640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec406b7b0 0x7f3ec40b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 -- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 msgr2=0x7f3ec415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 0x7f3ec415f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 -- 172.21.15.3:0/2640637792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec406b7b0 msgr2=0x7f3ec40b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec406b7b0 0x7f3ec40b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec8cf7640 1 -- 172.21.15.3:0/2640637792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ebc009b30 con 0x7f3ec40b3310 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.188+0000 7f3ec9cf9640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 0x7f3ec415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3ec8cf7640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000e980 tx=0x7f3ec000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0009670 con 0x7f3ec40b3310 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ec000ca00 con 0x7f3ec40b3310 2024-03-24T10:33:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec41f3f90 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000cd90 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec41f4550 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3ec0005990 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.189+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f3ec0013a60 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.190+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec41f4e00 con 0x7f3ec40b3310 2024-03-24T10:33:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.192+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3ec41f4e00 con 0x7f3ec40b3310 2024-03-24T10:33:32.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.266+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b38bb608-40ac-4f3f-9ece-e4d002045f3d", "id": 6} v 0) -- 0x7f3ec4069650 con 0x7f3ec40b3310 2024-03-24T10:33:32.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.269+0000 7f3eb5ffb640 1 -- 172.21.15.3:0/2640637792 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b38bb608-40ac-4f3f-9ece-e4d002045f3d", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f3ec0021060 con 0x7f3ec40b3310 2024-03-24T10:33:32.270 INFO:teuthology.orchestra.run.smithi003.stdout:6 2024-03-24T10:33:32.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 msgr2=0x7f3ec4161270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000e980 tx=0x7f3ec000ee40 comp rx=0 tx=0).stop 2024-03-24T10:33:32.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 shutdown_connections 2024-03-24T10:33:32.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ec415e320 0x7f3ec415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ec40b3310 0x7f3ec4161270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 --2- 172.21.15.3:0/2640637792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec406b7b0 0x7f3ec40b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.270+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 >> 172.21.15.3:0/2640637792 conn(0x7f3ec40b2820 msgr2=0x7f3ec40aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.271+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 shutdown_connections 2024-03-24T10:33:32.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.271+0000 7f3ecb75c640 1 -- 172.21.15.3:0/2640637792 wait complete. 2024-03-24T10:33:32.275 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.273+0000 7f1d1d0fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:33:32.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 001d918f-3181-4d41-82e5-6e03f6410e36 7 2024-03-24T10:33:32.290 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.290+0000 7f65dd248040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:33:32.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.383+0000 7fb78f5f7640 1 Processor -- start 2024-03-24T10:33:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.383+0000 7fb78f5f7640 1 -- start start 2024-03-24T10:33:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78f5f7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb78815e320 0x7fb7880cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78f5f7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb788072620 con 0x7fb78806b7b0 2024-03-24T10:33:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78f5f7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7880d02b0 con 0x7fb7880b3310 2024-03-24T10:33:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78f5f7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb78815e700 con 0x7fb78815e320 2024-03-24T10:33:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78cb92640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb7880b3310 0x7fb78806b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47578/0 (socket says 172.21.15.3:47578) 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.384+0000 7fb78cb92640 1 -- 172.21.15.3:0/4018850212 learned_addr learned my addr 172.21.15.3:0/4018850212 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb78d393640 1 --2- 172.21.15.3:0/4018850212 >> v2:172.21.15.1:3300/0 conn(0x7fb78815e320 0x7fb7880cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310209871 0 0) 0x7fb7880d02b0 con 0x7fb7880b3310 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb774003520 con 0x7fb7880b3310 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 856125518 0 0) 0x7fb788072620 con 0x7fb78806b7b0 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7880d02b0 con 0x7fb78806b7b0 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb78d393640 1 -- 172.21.15.3:0/4018850212 >> v1:172.21.15.3:6789/0 conn(0x7fb7880b3310 legacy=0x7fb78806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb78d393640 1 -- 172.21.15.3:0/4018850212 >> v1:172.21.15.1:6789/0 conn(0x7fb78806b7b0 legacy=0x7fb7880ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.385+0000 7fb78d393640 1 -- 172.21.15.3:0/4018850212 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7880cfcf0 con 0x7fb78815e320 2024-03-24T10:33:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb78d393640 1 --2- 172.21.15.3:0/4018850212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7880cf660 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7840097a0 tx=0x7fb78402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cba8fcce5ab65b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784031830 con 0x7fb78815e320 2024-03-24T10:33:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb784031990 con 0x7fb78815e320 2024-03-24T10:33:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb77ffff640 1 -- 172.21.15.3:0/4018850212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784031d00 con 0x7fb78815e320 2024-03-24T10:33:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb78f5f7640 1 -- 172.21.15.3:0/4018850212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 msgr2=0x7fb7880cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.386+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/4018850212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7880cf660 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7840097a0 tx=0x7fb78402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.387+0000 7fb78f5f7640 1 -- 172.21.15.3:0/4018850212 shutdown_connections 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.387+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/4018850212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7880cf660 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.387+0000 7fb78f5f7640 1 -- 172.21.15.3:0/4018850212 >> 172.21.15.3:0/4018850212 conn(0x7fb7880b2820 msgr2=0x7fb7880b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.387+0000 7fb78f5f7640 1 -- 172.21.15.3:0/4018850212 shutdown_connections 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.387+0000 7fb78f5f7640 1 -- 172.21.15.3:0/4018850212 wait complete. 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 Processor -- start 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 -- start start 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 0x7fb788159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7881f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb78815a8d0 con 0x7fb78815e320 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb7880c6740 con 0x7fb7880b3310 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.388+0000 7fb78f5f7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb7880c5b30 con 0x7fb78806b7b0 2024-03-24T10:33:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78d393640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 0x7fb788159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50612/0 (socket says 172.21.15.3:50612) 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 -- 172.21.15.3:0/3494694112 learned_addr learned my addr 172.21.15.3:0/3494694112 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78db94640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7881f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 -- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 msgr2=0x7fb788159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 0x7fb788159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 -- 172.21.15.3:0/3494694112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 msgr2=0x7fb7881f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7881f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 -- 172.21.15.3:0/3494694112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb784009400 con 0x7fb7880b3310 2024-03-24T10:33:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78d393640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 0x7fb788159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78cb92640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb77800c500 tx=0x7fb77800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778005ce0 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb778005e40 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7880c5e10 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.389+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7780072f0 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.390+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7880c63d0 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.390+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb778007450 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.390+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fb778012a60 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.390+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb788069650 con 0x7fb7880b3310 2024-03-24T10:33:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.394+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb778017070 con 0x7fb7880b3310 2024-03-24T10:33:32.413 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.413+0000 7f1d1d0fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:33:32.435 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.434+0000 7f65dd248040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:33:32.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.468+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd new", "uuid": "001d918f-3181-4d41-82e5-6e03f6410e36", "id": 7} v 0) -- 0x7fb78804f470 con 0x7fb7880b3310 2024-03-24T10:33:32.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.476+0000 7fb77dffb640 1 -- 172.21.15.3:0/3494694112 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "001d918f-3181-4d41-82e5-6e03f6410e36", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fb778012400 con 0x7fb7880b3310 2024-03-24T10:33:32.477 INFO:teuthology.orchestra.run.smithi003.stdout:7 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 msgr2=0x7fb7880c6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb77800c500 tx=0x7fb77800c9c0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 shutdown_connections 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb78815e320 0x7fb7881f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7880b3310 0x7fb7880c6bd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 --2- 172.21.15.3:0/3494694112 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb78806b7b0 0x7fb788159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 >> 172.21.15.3:0/3494694112 conn(0x7fb7880b2820 msgr2=0x7fb78806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 shutdown_connections 2024-03-24T10:33:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:33:32.477+0000 7fb78f5f7640 1 -- 172.21.15.3:0/3494694112 wait complete. 2024-03-24T10:33:32.488 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-24T10:33:32.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-24T10:33:32.490 INFO:tasks.ceph.osd.0:Started 2024-03-24T10:33:32.490 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-24T10:33:32.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-24T10:33:32.499 INFO:tasks.ceph.osd.1:Started 2024-03-24T10:33:32.500 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-24T10:33:32.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-24T10:33:32.502 INFO:tasks.ceph.osd.2:Started 2024-03-24T10:33:32.502 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-24T10:33:32.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-24T10:33:32.506 INFO:tasks.ceph.osd.3:Started 2024-03-24T10:33:32.506 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-24T10:33:32.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-24T10:33:32.530 INFO:tasks.ceph.osd.4:Started 2024-03-24T10:33:32.531 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-24T10:33:32.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-24T10:33:32.532 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.531+0000 7f1d1d0fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:33:32.533 INFO:tasks.ceph.osd.5:Started 2024-03-24T10:33:32.533 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-24T10:33:32.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-24T10:33:32.536 INFO:tasks.ceph.osd.6:Started 2024-03-24T10:33:32.536 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-24T10:33:32.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-24T10:33:32.540 INFO:tasks.ceph.osd.7:Started 2024-03-24T10:33:32.540 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:32.552 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.552+0000 7f65dd248040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:33:32.595 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.595+0000 7f1d1d0fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:33:32.627 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.627+0000 7f65dd248040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:33:32.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.675+0000 7f46f6a8f640 1 Processor -- start 2024-03-24T10:33:32.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.675+0000 7f46f6a8f640 1 -- start start 2024-03-24T10:33:32.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f6a8f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f46f0145260 0x7f46f006e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f6a8f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f006e8a0 0x7f46f006b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f6a8f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f00cfa10 con 0x7f46f0144b90 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f6a8f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f46f006fbc0 con 0x7f46f0145260 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f6a8f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f46f0144f70 con 0x7f46f006e8a0 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46effff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f46f0145260 0x7f46f006e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f482b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f46f0144b90 0x7f46f00cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:53812/0 (socket says 172.21.15.1:53812) 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f482b640 1 -- 172.21.15.1:0/3504343639 learned_addr learned my addr 172.21.15.1:0/3504343639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46f502c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f006e8a0 0x7f46f006b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46effff640 1 -- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f006e8a0 msgr2=0x7f46f006b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46effff640 1 --2- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f006e8a0 0x7f46f006b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46effff640 1 -- 172.21.15.1:0/3504343639 >> v1:172.21.15.3:6789/0 conn(0x7f46f0144b90 legacy=0x7f46f00cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46effff640 1 -- 172.21.15.1:0/3504343639 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f006de80 con 0x7f46f0145260 2024-03-24T10:33:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.676+0000 7f46ef7fe640 1 -- 172.21.15.1:0/3504343639 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704860173 0 0) 0x7f46f00cfa10 con 0x7f46f0144b90 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.677+0000 7f46effff640 1 --2- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f0145260 0x7f46f006e360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7b5728b5412f665 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.677+0000 7f46ef7fe640 1 -- 172.21.15.1:0/3504343639 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc031830 con 0x7f46f0145260 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.677+0000 7f46ef7fe640 1 -- 172.21.15.1:0/3504343639 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46dc031990 con 0x7f46f0145260 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.677+0000 7f46ef7fe640 1 -- 172.21.15.1:0/3504343639 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc031d20 con 0x7f46f0145260 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 -- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f0145260 msgr2=0x7f46f006e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f0145260 0x7f46f006e360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 -- 172.21.15.1:0/3504343639 shutdown_connections 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f006e8a0 0x7f46f006b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/3504343639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f0145260 0x7f46f006e360 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 -- 172.21.15.1:0/3504343639 >> 172.21.15.1:0/3504343639 conn(0x7f46f00b2890 msgr2=0x7f46f00b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 -- 172.21.15.1:0/3504343639 shutdown_connections 2024-03-24T10:33:32.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.678+0000 7f46f6a8f640 1 -- 172.21.15.1:0/3504343639 wait complete. 2024-03-24T10:33:32.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.679+0000 7f46f6a8f640 1 Processor -- start 2024-03-24T10:33:32.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.679+0000 7f46f6a8f640 1 -- start start 2024-03-24T10:33:32.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 0x7f46f00ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46f0145260 0x7f46f01f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:32.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f46f00cdac0 con 0x7f46f0145260 2024-03-24T10:33:32.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f46f014e370 con 0x7f46f006e8a0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f6a8f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f46f014e650 con 0x7f46f0144b90 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46effff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 0x7f46f00ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46effff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 0x7f46f00ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:44774/0 (socket says 172.21.15.1:44774) 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f502c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46f0145260 0x7f46f01f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:50400/0 (socket says 172.21.15.1:50400) 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46effff640 1 -- 172.21.15.1:0/2495148892 learned_addr learned my addr 172.21.15.1:0/2495148892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 -- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 msgr2=0x7f46f00ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 0x7f46f00ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 -- 172.21.15.1:0/2495148892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46f0145260 msgr2=0x7f46f01f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46f0145260 0x7f46f01f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.680+0000 7f46f482b640 1 -- 172.21.15.1:0/2495148892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc009400 con 0x7f46f006e8a0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46f482b640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f46e400c450 tx=0x7f46e400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:32.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e40074e0 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e4007640 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f014e930 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4004a20 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f01f5050 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f46e4004b80 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.681+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f46e4004e30 con 0x7f46f006e8a0 2024-03-24T10:33:32.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.682+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f014ee50 con 0x7f46f006e8a0 2024-03-24T10:33:32.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.686+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f46f014ee50 con 0x7f46f006e8a0 2024-03-24T10:33:32.736 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.736+0000 7f1d1d0fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:33:32.746 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.746+0000 7f65dd248040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:33:32.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.760+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46f004f4e0 con 0x7f46f006e8a0 2024-03-24T10:33:32.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.761+0000 7f46ed7fa640 1 -- 172.21.15.1:0/2495148892 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f46e4018070 con 0x7f46f006e8a0 2024-03-24T10:33:32.763 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:32.763 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:32.470118+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 msgr2=0x7f46f00c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f46e400c450 tx=0x7f46e400c910 comp rx=0 tx=0).stop 2024-03-24T10:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 shutdown_connections 2024-03-24T10:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46f0145260 0x7f46f01f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f46f0144b90 0x7f46f00ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 --2- 172.21.15.1:0/2495148892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46f006e8a0 0x7f46f00c9fb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 >> 172.21.15.1:0/2495148892 conn(0x7f46f00b2890 msgr2=0x7f46f00d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 shutdown_connections 2024-03-24T10:33:32.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:32.762+0000 7f46f6a8f640 1 -- 172.21.15.1:0/2495148892 wait complete. 2024-03-24T10:33:32.777 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-24T10:33:32.777 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-24T10:33:32.841 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:32.841+0000 7f1d1d0fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:33:32.845 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:32.845+0000 7f65dd248040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:33:33.001 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:33.000+0000 7f1d1d0fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:33:33.007 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:33.005+0000 7f65dd248040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:33:33.044 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:33.044+0000 7f1d1d0fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:33:33.049 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:33.049+0000 7f65dd248040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:33:33.096 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-24T10:33:33.096+0000 7f7ac7e11640 -1 Falling back to public interface 2024-03-24T10:33:33.113 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-24T10:33:33.113+0000 7fa049de1640 -1 Falling back to public interface 2024-03-24T10:33:33.121 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:33.121+0000 7fe005466640 -1 Falling back to public interface 2024-03-24T10:33:33.122 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-24T10:33:33.121+0000 7ff7c3e11640 -1 Falling back to public interface 2024-03-24T10:33:33.122 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-24T10:33:33.122+0000 7ff7c26fe640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6818/0: (98) Address already in use 2024-03-24T10:33:33.122 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:33.122+0000 7fe004415640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2024-03-24T10:33:33.127 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:33.124+0000 7fe004415640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6829/0: (98) Address already in use 2024-03-24T10:33:33.135 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-03-24T10:33:33.135+0000 7f5a334ae640 -1 Falling back to public interface 2024-03-24T10:33:33.138 INFO:tasks.ceph.osd.6.smithi003.stderr:2024-03-24T10:33:33.138+0000 7f71b4ea4640 -1 Falling back to public interface 2024-03-24T10:33:33.144 INFO:tasks.ceph.osd.7.smithi003.stderr:2024-03-24T10:33:33.144+0000 7f7be820e640 -1 Falling back to public interface 2024-03-24T10:33:33.157 INFO:tasks.ceph.osd.5.smithi003.stderr:2024-03-24T10:33:33.157+0000 7f4a5ba21640 -1 Falling back to public interface 2024-03-24T10:33:33.206 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:33.205+0000 7f1d1d0fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:33:33.211 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:33.211+0000 7f65dd248040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:33:33.588 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:33.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.679+0000 7f793c76c640 1 Processor -- start 2024-03-24T10:33:33.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.679+0000 7f793c76c640 1 -- start start 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793c76c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 0x7f79340d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793c76c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79340cd030 0x7f79340cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793c76c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79340d0b20 con 0x7f79340d0200 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793c76c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f79340ccc10 con 0x7f79340cd030 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793c76c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f793406dc00 con 0x7f79340cfa00 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79340cd030 0x7f79340cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793a508640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f79340d0200 0x7f793406d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46094/0 (socket says 172.21.15.1:46094) 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79340cd030 0x7f79340cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51210/0 (socket says 172.21.15.1:51210) 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793a508640 1 -- 172.21.15.1:0/2231510244 learned_addr learned my addr 172.21.15.1:0/2231510244 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f7939d07640 1 --2- 172.21.15.1:0/2231510244 >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 0x7f79340d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 -- 172.21.15.1:0/2231510244 >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 msgr2=0x7f79340d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:33.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 --2- 172.21.15.1:0/2231510244 >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 0x7f79340d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 -- 172.21.15.1:0/2231510244 >> v1:172.21.15.1:6789/0 conn(0x7f79340d0200 legacy=0x7f793406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f793ad09640 1 -- 172.21.15.1:0/2231510244 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79340cc270 con 0x7f79340cd030 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f7939506640 1 -- 172.21.15.1:0/2231510244 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932146080 0 0) 0x7f79340d0b20 con 0x7f79340d0200 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.680+0000 7f7939d07640 1 --2- 172.21.15.1:0/2231510244 >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 0x7f79340d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f793ad09640 1 --2- 172.21.15.1:0/2231510244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f79340cd4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7934071e10 tx=0x7f792c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529e8789812fe597 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f7939506640 1 -- 172.21.15.1:0/2231510244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c02cc80 con 0x7f79340cd030 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f7939506640 1 -- 172.21.15.1:0/2231510244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792c02cde0 con 0x7f79340cd030 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f7939506640 1 -- 172.21.15.1:0/2231510244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c034660 con 0x7f79340cd030 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f793c76c640 1 -- 172.21.15.1:0/2231510244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 msgr2=0x7f79340cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.681+0000 7f793c76c640 1 --2- 172.21.15.1:0/2231510244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f79340cd4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7934071e10 tx=0x7f792c02bed0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 -- 172.21.15.1:0/2231510244 shutdown_connections 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 --2- 172.21.15.1:0/2231510244 >> v2:172.21.15.3:3300/0 conn(0x7f79340cfa00 0x7f79340d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 --2- 172.21.15.1:0/2231510244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f79340cd4c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 -- 172.21.15.1:0/2231510244 >> 172.21.15.1:0/2231510244 conn(0x7f79340d1c20 msgr2=0x7f79340d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:33.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 -- 172.21.15.1:0/2231510244 shutdown_connections 2024-03-24T10:33:33.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.682+0000 7f793c76c640 1 -- 172.21.15.1:0/2231510244 wait complete. 2024-03-24T10:33:33.699 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.683+0000 7f793c76c640 1 Processor -- start 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.683+0000 7f793c76c640 1 -- start start 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79340cfa00 0x7f79340c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f79340d0200 0x7f79340af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f79340ca4f0 con 0x7f79340cd030 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f79340ca7d0 con 0x7f79340cfa00 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f79340caab0 con 0x7f79340d0200 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51224/0 (socket says 172.21.15.1:51224) 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 -- 172.21.15.1:0/2884250921 learned_addr learned my addr 172.21.15.1:0/2884250921 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:33.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f7939d07640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79340cfa00 0x7f79340c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793ad09640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f79340d0200 0x7f79340af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 -- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f79340d0200 msgr2=0x7f79340af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f79340d0200 0x7f79340af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 -- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79340cfa00 msgr2=0x7f79340c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79340cfa00 0x7f79340c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 -- 172.21.15.1:0/2884250921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792c009a70 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793a508640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f793000c430 tx=0x7f793000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300074e0 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7930007660 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7930004a20 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79340cad90 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.684+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79340c5eb0 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.685+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7930004b80 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.685+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7930013ae0 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.685+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7934075330 con 0x7f79340cd030 2024-03-24T10:33:33.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.687+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7930013d40 con 0x7f79340cd030 2024-03-24T10:33:33.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.749+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79340b7980 con 0x7f79340cd030 2024-03-24T10:33:33.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.750+0000 7f792affd640 1 -- 172.21.15.1:0/2884250921 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7930013410 con 0x7f79340cd030 2024-03-24T10:33:33.750 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-24T10:33:33.750 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:32.470118+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:33.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 msgr2=0x7f7934159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:33.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f793000c430 tx=0x7f793000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 shutdown_connections 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f79340d0200 0x7f79340af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79340cfa00 0x7f79340c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 --2- 172.21.15.1:0/2884250921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79340cd030 0x7f7934159ef0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 >> 172.21.15.1:0/2884250921 conn(0x7f79340d1c20 msgr2=0x7f793406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 shutdown_connections 2024-03-24T10:33:33.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:33.751+0000 7f793c76c640 1 -- 172.21.15.1:0/2884250921 wait complete. 2024-03-24T10:33:33.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T10:33:34.035 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:34.035+0000 7f1d1d0fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:33:34.054 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:34.054+0000 7f65dd248040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:33:34.323 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:34.323+0000 7f1d1d0fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:33:34.342 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:34.342+0000 7f65dd248040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:33:34.376 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:34.376+0000 7f1d1d0fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:33:34.395 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:34.395+0000 7f65dd248040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:33:34.494 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:34.494+0000 7f1d1d0fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:33:34.516 INFO:tasks.ceph.mgr.x.smithi003.stderr:2024-03-24T10:33:34.516+0000 7f65dd248040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:33:35.521 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:35.521+0000 7f1cbe570640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T10:33:35.522 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-24T10:33:35.521+0000 7f1cbe570640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T10:33:36.572 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-24T10:33:36.571+0000 7f7ac7e11640 -1 osd.1 0 log_to_monitors true 2024-03-24T10:33:36.598 INFO:tasks.ceph.osd.6.smithi003.stderr:2024-03-24T10:33:36.598+0000 7f71b4ea4640 -1 osd.6 0 log_to_monitors true 2024-03-24T10:33:36.624 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-24T10:33:36.624+0000 7ff7c3e11640 -1 osd.3 0 log_to_monitors true 2024-03-24T10:33:36.629 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-24T10:33:36.628+0000 7fa049de1640 -1 osd.0 0 log_to_monitors true 2024-03-24T10:33:36.633 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:36.631+0000 7fe005466640 -1 osd.2 0 log_to_monitors true 2024-03-24T10:33:36.650 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-03-24T10:33:36.650+0000 7f5a334ae640 -1 osd.4 0 log_to_monitors true 2024-03-24T10:33:36.665 INFO:tasks.ceph.osd.7.smithi003.stderr:2024-03-24T10:33:36.665+0000 7f7be820e640 -1 osd.7 0 log_to_monitors true 2024-03-24T10:33:36.675 INFO:tasks.ceph.osd.5.smithi003.stderr:2024-03-24T10:33:36.674+0000 7f4a5ba21640 -1 osd.5 0 log_to_monitors true 2024-03-24T10:33:39.975 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:39.975+0000 7fe000bd9640 -1 osd.2 0 waiting for initial osdmap 2024-03-24T10:33:39.989 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-24T10:33:39.987+0000 7ff7beec2640 -1 osd.3 0 waiting for initial osdmap 2024-03-24T10:33:39.989 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-24T10:33:39.988+0000 7fdff920b640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:39.990 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-24T10:33:39.990+0000 7fa046d8c640 -1 osd.0 0 waiting for initial osdmap 2024-03-24T10:33:39.997 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-24T10:33:39.997+0000 7ff7b74f4640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.002 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-24T10:33:40.002+0000 7fa03db86640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.005 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-24T10:33:40.005+0000 7f7ac2fe1640 -1 osd.1 0 waiting for initial osdmap 2024-03-24T10:33:40.017 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-24T10:33:40.017+0000 7f7abb613640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.568 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:40.615 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-03-24T10:33:40.613+0000 7f5a2ec21640 -1 osd.4 0 waiting for initial osdmap 2024-03-24T10:33:40.617 INFO:tasks.ceph.osd.5.smithi003.stderr:2024-03-24T10:33:40.617+0000 7f4a56bd4640 -1 osd.5 0 waiting for initial osdmap 2024-03-24T10:33:40.623 INFO:tasks.ceph.osd.7.smithi003.stderr:2024-03-24T10:33:40.623+0000 7f7be33b0640 -1 osd.7 0 waiting for initial osdmap 2024-03-24T10:33:40.626 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-03-24T10:33:40.626+0000 7f5a27253640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.636 INFO:tasks.ceph.osd.6.smithi003.stderr:2024-03-24T10:33:40.628+0000 7f71b1e4f640 -1 osd.6 0 waiting for initial osdmap 2024-03-24T10:33:40.637 INFO:tasks.ceph.osd.5.smithi003.stderr:2024-03-24T10:33:40.633+0000 7f4a4f206640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.637 INFO:tasks.ceph.osd.7.smithi003.stderr:2024-03-24T10:33:40.633+0000 7f7bdb9e2640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.649 INFO:tasks.ceph.osd.6.smithi003.stderr:2024-03-24T10:33:40.648+0000 7f71a8c49640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:40.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.665+0000 7fb2834fb640 1 Processor -- start 2024-03-24T10:33:40.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 -- start start 2024-03-24T10:33:40.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb27c0d1a40 0x7fb27c0b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb27c0d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb27c0b5060 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb27c0d1080 con 0x7fb27c0d1a40 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb2834fb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb27c0d1ed0 con 0x7fb27c0d06b0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb281297640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb27c0d2cc0 0x7fb27c0d4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45522/0 (socket says 172.21.15.1:45522) 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.666+0000 7fb281297640 1 -- 172.21.15.1:0/3186661375 learned_addr learned my addr 172.21.15.1:0/3186661375 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 --2- 172.21.15.1:0/3186661375 >> v2:172.21.15.3:3300/0 conn(0x7fb27c0d1a40 0x7fb27c0b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb281a98640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb27c0d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838718014 0 0) 0x7fb27c0b5060 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb264003520 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3260041281 0 0) 0x7fb264003520 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 msgr2=0x7fb27c0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb27c0d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 >> v2:172.21.15.3:3300/0 conn(0x7fb27c0d1a40 msgr2=0x7fb27c0b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 --2- 172.21.15.1:0/3186661375 >> v2:172.21.15.3:3300/0 conn(0x7fb27c0d1a40 0x7fb27c0b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c0cea80 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fb274001670 con 0x7fb27c0d2cc0 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 >> v1:172.21.15.1:6789/0 conn(0x7fb27c0d2cc0 legacy=0x7fb27c0d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb264003970 0x7fb264003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2640047c0 0x7fb264004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb264005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c0d1080 con 0x7fb2640047c0 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c0d1ed0 con 0x7fb264003970 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb26400b980 con 0x7fb27c0d06b0 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2640047c0 0x7fb264004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb281297640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb264003970 0x7fb264003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb281a98640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb264005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 -- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 msgr2=0x7fb264005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb27c0d06b0 0x7fb264005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 -- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb264003970 msgr2=0x7fb264003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb264003970 0x7fb264003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.667+0000 7fb280a96640 1 -- 172.21.15.1:0/3186661375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb264003520 con 0x7fb2640047c0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.668+0000 7fb280a96640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2640047c0 0x7fb264004c40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb270009af0 tx=0x7fb2640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01023ae06f01244 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.668+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27002c710 con 0x7fb2640047c0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.668+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27002c870 con 0x7fb2640047c0 2024-03-24T10:33:40.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.668+0000 7fb27bfff640 1 -- 172.21.15.1:0/3186661375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27002cc00 con 0x7fb2640047c0 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 -- 172.21.15.1:0/3186661375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2640047c0 msgr2=0x7fb264004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 --2- 172.21.15.1:0/3186661375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2640047c0 0x7fb264004c40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb270009af0 tx=0x7fb2640057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb281a98640 1 -- 172.21.15.1:0/3186661375 reap_dead start 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 -- 172.21.15.1:0/3186661375 shutdown_connections 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 -- 172.21.15.1:0/3186661375 >> 172.21.15.1:0/3186661375 conn(0x7fb27c09d1a0 msgr2=0x7fb27c0d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 -- 172.21.15.1:0/3186661375 shutdown_connections 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.669+0000 7fb2834fb640 1 -- 172.21.15.1:0/3186661375 wait complete. 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.670+0000 7fb2834fb640 1 Processor -- start 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.670+0000 7fb2834fb640 1 -- start start 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2640047c0 0x7fb27c0cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 0x7fb27c161c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c0cde60 con 0x7fb27c1521a0 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c153300 con 0x7fb27c152620 2024-03-24T10:33:40.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb2834fb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb27c0cdb80 con 0x7fb2640047c0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35052/0 (socket says 172.21.15.1:35052) 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 -- 172.21.15.1:0/4066598402 learned_addr learned my addr 172.21.15.1:0/4066598402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb281a98640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 0x7fb27c161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb281297640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2640047c0 0x7fb27c0cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 -- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2640047c0 msgr2=0x7fb27c0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2640047c0 0x7fb27c0cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 -- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 msgr2=0x7fb27c161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 0x7fb27c161c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 -- 172.21.15.1:0/4066598402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c152cb0 con 0x7fb27c1521a0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb281a98640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 0x7fb27c161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.671+0000 7fb280a96640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb270037040 tx=0x7fb270004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.672+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270003810 con 0x7fb27c1521a0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.672+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb270003970 con 0x7fb27c1521a0 2024-03-24T10:33:40.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.672+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c152f90 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.672+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270003d20 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.672+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c0cc130 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.673+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c152b10 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.673+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fb270040450 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.674+0000 7fb279ffb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 0x7fb26806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.674+0000 7fb281297640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 0x7fb26806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.674+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fb2700ab0d0 con 0x7fb27c1521a0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.674+0000 7fb281297640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 0x7fb26806dc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb274000fb0 tx=0x7fb274009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:40.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.678+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb27c152b10 con 0x7fb27c1521a0 2024-03-24T10:33:40.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.860+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb27c0cc130 con 0x7fb27c1521a0 2024-03-24T10:33:40.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.861+0000 7fb279ffb640 1 -- 172.21.15.1:0/4066598402 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fb27c0cc130 con 0x7fb27c1521a0 2024-03-24T10:33:40.861 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-24T10:33:40.861 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":12,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:38.522435+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:40.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 msgr2=0x7fb26806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 0x7fb26806dc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb274000fb0 tx=0x7fb274009040 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 msgr2=0x7fb27c0cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb270037040 tx=0x7fb270004730 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 shutdown_connections 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb26806b780 0x7fb26806dc40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb27c152620 0x7fb27c161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb27c1521a0 0x7fb27c0cba10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 --2- 172.21.15.1:0/4066598402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2640047c0 0x7fb27c0cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.862+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 >> 172.21.15.1:0/4066598402 conn(0x7fb27c09d1a0 msgr2=0x7fb27c161240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.863+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 shutdown_connections 2024-03-24T10:33:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:40.863+0000 7fb2834fb640 1 -- 172.21.15.1:0/4066598402 wait complete. 2024-03-24T10:33:40.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T10:33:47.679 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:47.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.780+0000 7f9f55023640 1 Processor -- start 2024-03-24T10:33:47.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.780+0000 7f9f55023640 1 -- start start 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.780+0000 7f9f55023640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 0x7f9f500b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f55023640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f500d4db0 con 0x7f9f5015e3d0 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f55023640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f500b3420 con 0x7f9f500d2b10 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f55023640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9f500b3700 con 0x7f9f500d0f80 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4ed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9f5015e3d0 0x7f9f500d2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45528/0 (socket says 172.21.15.1:45528) 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4ed9d640 1 -- 172.21.15.1:0/1506522229 learned_addr learned my addr 172.21.15.1:0/1506522229 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4f59e640 1 --2- 172.21.15.1:0/1506522229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 0x7f9f500b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526213430 0 0) 0x7f9f40001110 con 0x7f9f5015e3d0 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f38003520 con 0x7f9f5015e3d0 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462584650 0 0) 0x7f9f500b3420 con 0x7f9f500d2b10 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f40001110 con 0x7f9f500d2b10 2024-03-24T10:33:47.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2428681927 0 0) 0x7f9f38003520 con 0x7f9f5015e3d0 2024-03-24T10:33:47.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4f59e640 1 -- 172.21.15.1:0/1506522229 >> v1:172.21.15.3:6789/0 conn(0x7f9f500d2b10 legacy=0x7f9f500b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4f59e640 1 -- 172.21.15.1:0/1506522229 >> v1:172.21.15.1:6789/0 conn(0x7f9f5015e3d0 legacy=0x7f9f500d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.781+0000 7f9f4f59e640 1 -- 172.21.15.1:0/1506522229 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f500b39e0 con 0x7f9f500d0f80 2024-03-24T10:33:47.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.782+0000 7f9f4f59e640 1 --2- 172.21.15.1:0/1506522229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 0x7f9f500b48f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009dc0 tx=0x7f9f4802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6ad4380b7d6362 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:47.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.782+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40001800 con 0x7f9f500d0f80 2024-03-24T10:33:47.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.782+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4802c910 con 0x7f9f500d0f80 2024-03-24T10:33:47.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.782+0000 7f9f4dd9b640 1 -- 172.21.15.1:0/1506522229 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4802cc80 con 0x7f9f500d0f80 2024-03-24T10:33:47.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 -- 172.21.15.1:0/1506522229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 msgr2=0x7f9f500b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 --2- 172.21.15.1:0/1506522229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 0x7f9f500b48f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009dc0 tx=0x7f9f4802bde0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 -- 172.21.15.1:0/1506522229 shutdown_connections 2024-03-24T10:33:47.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 --2- 172.21.15.1:0/1506522229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f500d0f80 0x7f9f500b48f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 -- 172.21.15.1:0/1506522229 >> 172.21.15.1:0/1506522229 conn(0x7f9f500d3e20 msgr2=0x7f9f500d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:47.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 -- 172.21.15.1:0/1506522229 shutdown_connections 2024-03-24T10:33:47.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.783+0000 7f9f55023640 1 -- 172.21.15.1:0/1506522229 wait complete. 2024-03-24T10:33:47.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.784+0000 7f9f55023640 1 Processor -- start 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.784+0000 7f9f55023640 1 -- start start 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 0x7f9f5014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f5015e3d0 0x7f9f500cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b6eb0 con 0x7f9f500d2b10 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b7190 con 0x7f9f500d0f80 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f55023640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9f500b7470 con 0x7f9f5015e3d0 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f4e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f4e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35090/0 (socket says 172.21.15.1:35090) 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f4e59c640 1 -- 172.21.15.1:0/3134248136 learned_addr learned my addr 172.21.15.1:0/3134248136 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f4f59e640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f5015e3d0 0x7f9f500cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:47.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.785+0000 7f9f4ed9d640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 0x7f9f5014f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 -- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f5015e3d0 msgr2=0x7f9f500cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f5015e3d0 0x7f9f500cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 -- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 msgr2=0x7f9f5014f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 0x7f9f5014f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 -- 172.21.15.1:0/3134248136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f48009a70 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4ed9d640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 0x7f9f5014f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f4e59c640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c490 tx=0x7f9f4400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f440052d0 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f44005430 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f500b7750 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44012870 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.786+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f5014f720 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.787+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f50068640 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.788+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9f440129d0 con 0x7f9f500d2b10 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.788+0000 7f9f377fe640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 0x7f9f3c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:47.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.788+0000 7f9f4ed9d640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 0x7f9f3c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:47.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.788+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9f44089910 con 0x7f9f500d2b10 2024-03-24T10:33:47.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.789+0000 7f9f4ed9d640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 0x7f9f3c06dd00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f40001370 tx=0x7f9f40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:47.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.793+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f44054710 con 0x7f9f500d2b10 2024-03-24T10:33:47.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.975+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f50075330 con 0x7f9f500d2b10 2024-03-24T10:33:47.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.976+0000 7f9f377fe640 1 -- 172.21.15.1:0/3134248136 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12752 (secure 0 0 0) 0x7f9f44057dd0 con 0x7f9f500d2b10 2024-03-24T10:33:47.977 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-24T10:33:47.977 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":17,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:45.015969+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.978+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 msgr2=0x7f9f3c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.978+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 0x7f9f3c06dd00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f40001370 tx=0x7f9f40009040 comp rx=0 tx=0).stop 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.978+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 msgr2=0x7f9f50160f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.978+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c490 tx=0x7f9f4400c950 comp rx=0 tx=0).stop 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.978+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 shutdown_connections 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9f3c06b840 0x7f9f3c06dd00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.981 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9f5015e3d0 0x7f9f500cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f500d2b10 0x7f9f50160f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 --2- 172.21.15.1:0/3134248136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f500d0f80 0x7f9f5014f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 >> 172.21.15.1:0/3134248136 conn(0x7f9f500d3e20 msgr2=0x7f9f500cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 shutdown_connections 2024-03-24T10:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-24T10:33:47.979+0000 7f9f55023640 1 -- 172.21.15.1:0/3134248136 wait complete. 2024-03-24T10:33:47.989 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-24T10:33:47.990 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-24T10:33:47.990 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T10:33:47.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T10:33:47.993 INFO:tasks.ceph.mds.a:Started 2024-03-24T10:33:47.993 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T10:33:47.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T10:33:47.996 INFO:tasks.ceph.mds.c:Started 2024-03-24T10:33:47.996 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T10:33:47.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T10:33:47.999 INFO:tasks.ceph.mds.b:Started 2024-03-24T10:33:48.000 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T10:33:48.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T10:33:48.002 INFO:tasks.ceph.mds.d:Started 2024-03-24T10:33:48.002 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-24T10:33:48.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:48.068 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-24T10:33:48.070 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-24T10:33:48.086 INFO:tasks.ceph.mds.d.smithi003.stdout:starting mds.d at 2024-03-24T10:33:48.086 INFO:tasks.ceph.mds.b.smithi003.stdout:starting mds.b at 2024-03-24T10:33:48.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.107+0000 7f3e81da1640 1 Processor -- start 2024-03-24T10:33:48.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.107+0000 7f3e81da1640 1 -- start start 2024-03-24T10:33:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e81da1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3e7c072480 0x7f3e7c06ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e81da1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 0x7f3e7c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e81da1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c0cbef0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e81da1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c15f330 con 0x7f3e7c0cb670 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3e7c15acf0 0x7f3e7c071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45534/0 (socket says 172.21.15.1:45534) 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7b7fe640 1 -- 172.21.15.1:0/2172717280 learned_addr learned my addr 172.21.15.1:0/2172717280 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7bfff640 1 --2- 172.21.15.1:0/2172717280 >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 0x7f3e7c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c15f7c0 con 0x7f3e7c072480 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7affd640 1 --2- 172.21.15.1:0/2172717280 >> v2:172.21.15.3:3300/0 conn(0x7f3e7c072480 0x7f3e7c06ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7a7fc640 1 -- 172.21.15.1:0/2172717280 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790052834 0 0) 0x7f3e7c0cbef0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7a7fc640 1 -- 172.21.15.1:0/2172717280 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e64003520 con 0x7f3e7c15acf0 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7affd640 1 -- 172.21.15.1:0/2172717280 >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 msgr2=0x7f3e7c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7affd640 1 --2- 172.21.15.1:0/2172717280 >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 0x7f3e7c06efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7affd640 1 -- 172.21.15.1:0/2172717280 >> v1:172.21.15.1:6789/0 conn(0x7f3e7c15acf0 legacy=0x7f3e7c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7affd640 1 -- 172.21.15.1:0/2172717280 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c15ea00 con 0x7f3e7c072480 2024-03-24T10:33:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.108+0000 7f3e7bfff640 1 --2- 172.21.15.1:0/2172717280 >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 0x7f3e7c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.109+0000 7f3e7affd640 1 --2- 172.21.15.1:0/2172717280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c072480 0x7f3e7c06ea70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0d1c70 tx=0x7f3e7002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aa954f09b6ba6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.109+0000 7f3e7a7fc640 1 -- 172.21.15.1:0/2172717280 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7002e070 con 0x7f3e7c072480 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.109+0000 7f3e7a7fc640 1 -- 172.21.15.1:0/2172717280 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e700315a0 con 0x7f3e7c072480 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.109+0000 7f3e7a7fc640 1 -- 172.21.15.1:0/2172717280 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70031930 con 0x7f3e7c072480 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c072480 msgr2=0x7f3e7c06ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2172717280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c072480 0x7f3e7c06ea70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0d1c70 tx=0x7f3e7002b840 comp rx=0 tx=0).stop 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 shutdown_connections 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2172717280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c072480 0x7f3e7c06ea70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2172717280 >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0cb670 0x7f3e7c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 >> 172.21.15.1:0/2172717280 conn(0x7f3e7c0d0b70 msgr2=0x7f3e7c0d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 shutdown_connections 2024-03-24T10:33:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.110+0000 7f3e81da1640 1 -- 172.21.15.1:0/2172717280 wait complete. 2024-03-24T10:33:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.111+0000 7f3e81da1640 1 Processor -- start 2024-03-24T10:33:48.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.111+0000 7f3e81da1640 1 -- start start 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3e7c072480 0x7f3e7c14b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c0cb670 0x7f3e7c14d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b39b0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b3c90 con 0x7f3e7c0cb670 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e81da1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c0b3f70 con 0x7f3e7c072480 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35122/0 (socket says 172.21.15.1:35122) 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 -- 172.21.15.1:0/2322412691 learned_addr learned my addr 172.21.15.1:0/2322412691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7b7fe640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3e7c072480 0x7f3e7c14b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7affd640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c0cb670 0x7f3e7c14d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 -- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3e7c072480 msgr2=0x7f3e7c14b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3e7c072480 0x7f3e7c14b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 -- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c0cb670 msgr2=0x7f3e7c14d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c0cb670 0x7f3e7c14d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 -- 172.21.15.1:0/2322412691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e74009a70 con 0x7f3e7c15acf0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.112+0000 7f3e7bfff640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400efc0 tx=0x7f3e74009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.113+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74009d70 con 0x7f3e7c15acf0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.113+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e70009400 con 0x7f3e7c15acf0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.113+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7400f040 con 0x7f3e7c15acf0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.113+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74003990 con 0x7f3e7c15acf0 2024-03-24T10:33:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.113+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c0b4530 con 0x7f3e7c15acf0 2024-03-24T10:33:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.114+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7c0b4720 con 0x7f3e7c15acf0 2024-03-24T10:33:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.114+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e740144a0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.114+0000 7f3e63fff640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 0x7f3e6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.114+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f3e7408ab80 con 0x7f3e7c15acf0 2024-03-24T10:33:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.114+0000 7f3e7b7fe640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 0x7f3e6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.115+0000 7f3e7b7fe640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 0x7f3e6806dcd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c000f60 tx=0x7f3e6c007d10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.119+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e740557f0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.300+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e7c04f4e0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.301+0000 7f3e63fff640 1 -- 172.21.15.1:0/2322412691 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12752 (secure 0 0 0) 0x7f3e74058eb0 con 0x7f3e7c15acf0 2024-03-24T10:33:48.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:48.302 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:45.015969+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 msgr2=0x7f3e6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 0x7f3e6806dcd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c000f60 tx=0x7f3e6c007d10 comp rx=0 tx=0).stop 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 msgr2=0x7f3e7c0c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400efc0 tx=0x7f3e74009f90 comp rx=0 tx=0).stop 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 shutdown_connections 2024-03-24T10:33:48.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e6806b810 0x7f3e6806dcd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15acf0 0x7f3e7c0c9ea0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e7c0cb670 0x7f3e7c14d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 --2- 172.21.15.1:0/2322412691 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3e7c072480 0x7f3e7c14b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 >> 172.21.15.1:0/2322412691 conn(0x7f3e7c0d0b70 msgr2=0x7f3e7c154b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 shutdown_connections 2024-03-24T10:33:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.303+0000 7f3e81da1640 1 -- 172.21.15.1:0/2322412691 wait complete. 2024-03-24T10:33:48.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:33:48.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:33:48.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.418+0000 7f262cc9d640 1 Processor -- start 2024-03-24T10:33:48.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.418+0000 7f262cc9d640 1 -- start start 2024-03-24T10:33:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f262cc9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26280b0100 0x7f26280ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f262cc9d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2628072760 con 0x7f262806b7b0 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f262cc9d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26280b05d0 con 0x7f2628158540 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f262cc9d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f262815f510 con 0x7f26280b0100 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f2626d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26280b0100 0x7f26280ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f261ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f262806b7b0 0x7f26280af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45540/0 (socket says 172.21.15.1:45540) 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f261ffff640 1 -- 172.21.15.1:0/1684553201 learned_addr learned my addr 172.21.15.1:0/1684553201 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f2626d9d640 1 --2- 172.21.15.1:0/1684553201 >> v2:172.21.15.1:3300/0 conn(0x7f26280b0100 0x7f26280ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35126/0 (socket says 172.21.15.1:35126) 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f2626d9d640 1 -- 172.21.15.1:0/1684553201 >> v1:172.21.15.3:6789/0 conn(0x7f2628158540 legacy=0x7f262806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f2626d9d640 1 -- 172.21.15.1:0/1684553201 >> v1:172.21.15.1:6789/0 conn(0x7f262806b7b0 legacy=0x7f26280af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.419+0000 7f2626d9d640 1 -- 172.21.15.1:0/1684553201 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2628071e30 con 0x7f26280b0100 2024-03-24T10:33:48.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.420+0000 7f2626d9d640 1 --2- 172.21.15.1:0/1684553201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26280b0100 0x7f26280ae950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2620009d90 tx=0x7f262002bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f2113e2ecd62a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.420+0000 7f2625d9b640 1 -- 172.21.15.1:0/1684553201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262002c990 con 0x7f26280b0100 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.420+0000 7f2625d9b640 1 -- 172.21.15.1:0/1684553201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262002caf0 con 0x7f26280b0100 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.420+0000 7f2625d9b640 1 -- 172.21.15.1:0/1684553201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262002ce80 con 0x7f26280b0100 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 -- 172.21.15.1:0/1684553201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26280b0100 msgr2=0x7f26280ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1684553201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26280b0100 0x7f26280ae950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2620009d90 tx=0x7f262002bc60 comp rx=0 tx=0).stop 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 -- 172.21.15.1:0/1684553201 shutdown_connections 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1684553201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26280b0100 0x7f26280ae950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 -- 172.21.15.1:0/1684553201 >> 172.21.15.1:0/1684553201 conn(0x7f26280b2820 msgr2=0x7f26280b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 -- 172.21.15.1:0/1684553201 shutdown_connections 2024-03-24T10:33:48.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.421+0000 7f262cc9d640 1 -- 172.21.15.1:0/1684553201 wait complete. 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.422+0000 7f262cc9d640 1 Processor -- start 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.422+0000 7f262cc9d640 1 -- start start 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 0x7f26281532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b0100 0x7f26281e8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26281f7350 con 0x7f2628158540 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f26281e8de0 con 0x7f26280b0100 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f26281e90c0 con 0x7f262806b7b0 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35142/0 (socket says 172.21.15.1:35142) 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 -- 172.21.15.1:0/1423024971 learned_addr learned my addr 172.21.15.1:0/1423024971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:48.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f261ffff640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b0100 0x7f26281e8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262659c640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 0x7f26281532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 -- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 msgr2=0x7f26281532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 0x7f26281532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 -- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b0100 msgr2=0x7f26281e8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b0100 0x7f26281e8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 -- 172.21.15.1:0/1423024971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2620009a70 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262659c640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 0x7f26281532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f2626d9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2620009d90 tx=0x7f2620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262003e610 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262003e790 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26281e93a0 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262003eb40 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.423+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26281e9960 con 0x7f2628158540 2024-03-24T10:33:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.425+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2628157910 con 0x7f2628158540 2024-03-24T10:33:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.425+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f262003b050 con 0x7f2628158540 2024-03-24T10:33:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.425+0000 7f261effd640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 0x7f260806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.425+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f26200ac860 con 0x7f2628158540 2024-03-24T10:33:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.425+0000 7f262659c640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 0x7f260806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.426+0000 7f262659c640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 0x7f260806dcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2618003eb0 tx=0x7f2618009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.429+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2620077680 con 0x7f2628158540 2024-03-24T10:33:48.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.609+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2628157910 con 0x7f2628158540 2024-03-24T10:33:48.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.609+0000 7f261effd640 1 -- 172.21.15.1:0/1423024971 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f262007ad40 con 0x7f2628158540 2024-03-24T10:33:48.610 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:33:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 msgr2=0x7f260806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 0x7f260806dcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2618003eb0 tx=0x7f2618009040 comp rx=0 tx=0).stop 2024-03-24T10:33:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 msgr2=0x7f262806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2620009d90 tx=0x7f2620004a40 comp rx=0 tx=0).stop 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 shutdown_connections 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f260806b810 0x7f260806dcd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2628158540 0x7f262806dea0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b0100 0x7f26281e8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 --2- 172.21.15.1:0/1423024971 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f262806b7b0 0x7f26281532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 >> 172.21.15.1:0/1423024971 conn(0x7f26280b2820 msgr2=0x7f2628161960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 shutdown_connections 2024-03-24T10:33:48.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.611+0000 7f262cc9d640 1 -- 172.21.15.1:0/1423024971 wait complete. 2024-03-24T10:33:48.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:33:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 Processor -- start 2024-03-24T10:33:48.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 -- start start 2024-03-24T10:33:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe24415c000 0x7fe2440b4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2440cd210 con 0x7fe24406ee20 2024-03-24T10:33:48.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2440cf8d0 con 0x7fe24406f2a0 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.723+0000 7fe24a083640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe2440cfc40 con 0x7fe24415c000 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe24406ee20 0x7fe2440d0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45544/0 (socket says 172.21.15.1:45544) 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 -- 172.21.15.1:0/2515373654 learned_addr learned my addr 172.21.15.1:0/2515373654 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2437fe640 1 --2- 172.21.15.1:0/2515373654 >> v2:172.21.15.3:3300/0 conn(0x7fe24415c000 0x7fe2440b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658946547 0 0) 0x7fe2440cd210 con 0x7fe24406ee20 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe22c003520 con 0x7fe24406ee20 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1932486750 0 0) 0x7fe22c003520 con 0x7fe24406ee20 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 >> v2:172.21.15.3:3300/0 conn(0x7fe24415c000 msgr2=0x7fe2440b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 --2- 172.21.15.1:0/2515373654 >> v2:172.21.15.3:3300/0 conn(0x7fe24415c000 0x7fe2440b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 >> v1:172.21.15.3:6789/0 conn(0x7fe24406f2a0 legacy=0x7fe2440ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2440cd460 con 0x7fe24406ee20 2024-03-24T10:33:48.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fe238001670 con 0x7fe24406ee20 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 >> v1:172.21.15.1:6789/0 conn(0x7fe24406ee20 legacy=0x7fe2440d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe22c003970 0x7fe22c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 0x7fe22c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c005180 0x7fe22c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440cfc40 con 0x7fe22c0047c0 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c00c120 con 0x7fe22c003970 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe22c00c2e0 con 0x7fe22c005180 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2437fe640 1 --2- 172.21.15.1:0/2515373654 >> v2:172.21.15.3:3300/0 conn(0x7fe24415c000 0x7fe2440b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 0x7fe22c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe243fff640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c005180 0x7fe22c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe2437fe640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe22c003970 0x7fe22c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 -- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c005180 msgr2=0x7fe22c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c005180 0x7fe22c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 -- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe22c003970 msgr2=0x7fe22c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.724+0000 7fe242ffd640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe22c003970 0x7fe22c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe243fff640 1 -- 172.21.15.1:0/2515373654 reap_dead start 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe242ffd640 1 -- 172.21.15.1:0/2515373654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22c003520 con 0x7fe22c0047c0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe242ffd640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 0x7fe22c004c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c240 tx=0x7fe22c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e860b320aa5172 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238005ad0 con 0x7fe22c0047c0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe238005c30 con 0x7fe22c0047c0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.725+0000 7fe2427fc640 1 -- 172.21.15.1:0/2515373654 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23802caf0 con 0x7fe22c0047c0 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 -- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 msgr2=0x7fe22c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 0x7fe22c004c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c240 tx=0x7fe22c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 -- 172.21.15.1:0/2515373654 shutdown_connections 2024-03-24T10:33:48.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 --2- 172.21.15.1:0/2515373654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe22c0047c0 0x7fe22c004c40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 -- 172.21.15.1:0/2515373654 >> 172.21.15.1:0/2515373654 conn(0x7fe2440b3770 msgr2=0x7fe2440b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 -- 172.21.15.1:0/2515373654 shutdown_connections 2024-03-24T10:33:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.726+0000 7fe24a083640 1 -- 172.21.15.1:0/2515373654 wait complete. 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.727+0000 7fe24a083640 1 Processor -- start 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.727+0000 7fe24a083640 1 -- start start 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 0x7fe2440aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2440c5790 0x7fe2440c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440af2a0 con 0x7fe2440b0340 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440af580 con 0x7fe2440c5790 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe24a083640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe2440af860 con 0x7fe22c0047c0 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35170/0 (socket says 172.21.15.1:35170) 2024-03-24T10:33:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 -- 172.21.15.1:0/3511992880 learned_addr learned my addr 172.21.15.1:0/3511992880 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe243fff640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2440c5790 0x7fe2440c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe2437fe640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 0x7fe2440aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 -- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 msgr2=0x7fe2440aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 0x7fe2440aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 -- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2440c5790 msgr2=0x7fe2440c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2440c5790 0x7fe2440c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe242ffd640 1 -- 172.21.15.1:0/3511992880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2440afb40 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.728+0000 7fe2437fe640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 0x7fe2440aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe242ffd640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c240 tx=0x7fe23802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23803c040 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe23802cce0 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2380072b0 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2440afe20 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.729+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2440cab20 con 0x7fe2440b0340 2024-03-24T10:33:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.730+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe24406b560 con 0x7fe2440b0340 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.730+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe238032070 con 0x7fe2440b0340 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.730+0000 7fe22bfff640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 0x7fe23006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.731+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe2380ab2b0 con 0x7fe2440b0340 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.731+0000 7fe2437fe640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 0x7fe23006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.731+0000 7fe2437fe640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 0x7fe23006dcd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe2440699a0 tx=0x7fe234008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.735+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe2380760d0 con 0x7fe2440b0340 2024-03-24T10:33:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.953+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe24404f4e0 con 0x7fe2440b0340 2024-03-24T10:33:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.953+0000 7fe22bfff640 1 -- 172.21.15.1:0/3511992880 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1386 (secure 0 0 0) 0x7fe238079790 con 0x7fe2440b0340 2024-03-24T10:33:48.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:48.955 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T10:33:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 msgr2=0x7fe23006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 0x7fe23006dcd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe2440699a0 tx=0x7fe234008040 comp rx=0 tx=0).stop 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 msgr2=0x7fe2440c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c240 tx=0x7fe23802e070 comp rx=0 tx=0).stop 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 shutdown_connections 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe23006b810 0x7fe23006dcd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2440c5790 0x7fe2440c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2440b0340 0x7fe2440c8f40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 --2- 172.21.15.1:0/3511992880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe22c0047c0 0x7fe2440aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 >> 172.21.15.1:0/3511992880 conn(0x7fe2440b3770 msgr2=0x7fe2440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 shutdown_connections 2024-03-24T10:33:48.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:48.956+0000 7fe24a083640 1 -- 172.21.15.1:0/3511992880 wait complete. 2024-03-24T10:33:50.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:33:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.084+0000 7fe7f62d4640 1 Processor -- start 2024-03-24T10:33:51.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.084+0000 7fe7f62d4640 1 -- start start 2024-03-24T10:33:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f62d4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 0x7fe7f00d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f62d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7f006d2d0 0x7fe7f015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f62d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f00b4c70 con 0x7fe7f006c770 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f62d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe7f00d1340 con 0x7fe7f006d2d0 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f62d4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe7f00b4f50 con 0x7fe7f00b74c0 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f4871640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe7f006d2d0 0x7fe7f015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7ef7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe7f006c770 0x7fe7f006cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39500/0 (socket says 172.21.15.1:39500) 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7ef7fe640 1 -- 172.21.15.1:0/1196314101 learned_addr learned my addr 172.21.15.1:0/1196314101 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7effff640 1 --2- 172.21.15.1:0/1196314101 >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 0x7fe7f00d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f4871640 1 -- 172.21.15.1:0/1196314101 >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 msgr2=0x7fe7f00d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f4871640 1 --2- 172.21.15.1:0/1196314101 >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 0x7fe7f00d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f4871640 1 -- 172.21.15.1:0/1196314101 >> v1:172.21.15.1:6789/0 conn(0x7fe7f006c770 legacy=0x7fe7f006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7f4871640 1 -- 172.21.15.1:0/1196314101 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f00b5230 con 0x7fe7f006d2d0 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7eeffd640 1 -- 172.21.15.1:0/1196314101 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530304048 0 0) 0x7fe7f00b4c70 con 0x7fe7f006c770 2024-03-24T10:33:51.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.085+0000 7fe7effff640 1 --2- 172.21.15.1:0/1196314101 >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 0x7fe7f00d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.086+0000 7fe7f4871640 1 --2- 172.21.15.1:0/1196314101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015d8f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7f006bba0 tx=0x7fe7e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47eaef1c027d269 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.086+0000 7fe7eeffd640 1 -- 172.21.15.1:0/1196314101 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e802cc80 con 0x7fe7f006d2d0 2024-03-24T10:33:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.086+0000 7fe7eeffd640 1 -- 172.21.15.1:0/1196314101 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e802cde0 con 0x7fe7f006d2d0 2024-03-24T10:33:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.086+0000 7fe7eeffd640 1 -- 172.21.15.1:0/1196314101 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8034660 con 0x7fe7f006d2d0 2024-03-24T10:33:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 -- 172.21.15.1:0/1196314101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006d2d0 msgr2=0x7fe7f015d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/1196314101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015d8f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe7f006bba0 tx=0x7fe7e802bed0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 -- 172.21.15.1:0/1196314101 shutdown_connections 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/1196314101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015d8f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/1196314101 >> v2:172.21.15.3:3300/0 conn(0x7fe7f00b74c0 0x7fe7f00d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 -- 172.21.15.1:0/1196314101 >> 172.21.15.1:0/1196314101 conn(0x7fe7f00d1c80 msgr2=0x7fe7f00d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 -- 172.21.15.1:0/1196314101 shutdown_connections 2024-03-24T10:33:51.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.087+0000 7fe7f62d4640 1 -- 172.21.15.1:0/1196314101 wait complete. 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.088+0000 7fe7f62d4640 1 Processor -- start 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.088+0000 7fe7f62d4640 1 -- start start 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe7f00b74c0 0x7fe7f015a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f00b2610 con 0x7fe7f006c770 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f015d150 con 0x7fe7f006d2d0 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f62d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f015c5d0 con 0x7fe7f00b74c0 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45130/0 (socket says 172.21.15.1:45130) 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 -- 172.21.15.1:0/379059427 learned_addr learned my addr 172.21.15.1:0/379059427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:51.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7ef7fe640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7f4871640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe7f00b74c0 0x7fe7f015a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 -- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe7f00b74c0 msgr2=0x7fe7f015a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe7f00b74c0 0x7fe7f015a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 -- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7f006d2d0 msgr2=0x7fe7f015e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.089+0000 7fe7effff640 1 -- 172.21.15.1:0/379059427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e8009a70 con 0x7fe7f006c770 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7effff640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400c7f0 tx=0x7fe7e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4007820 con 0x7fe7f006c770 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e4007980 con 0x7fe7f006c770 2024-03-24T10:33:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f015c8b0 con 0x7fe7f006c770 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4007cf0 con 0x7fe7f006c770 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.090+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f015ce70 con 0x7fe7f006c770 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.091+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f004f4e0 con 0x7fe7f006c770 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.091+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe7e40113f0 con 0x7fe7f006c770 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.092+0000 7fe7ecff9640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 0x7fe7dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.092+0000 7fe7ef7fe640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 0x7fe7dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.092+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe7e408a0d0 con 0x7fe7f006c770 2024-03-24T10:33:51.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.092+0000 7fe7ef7fe640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 0x7fe7dc06dcd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007fd0 tx=0x7fe7e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.096+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe7e4054ed0 con 0x7fe7f006c770 2024-03-24T10:33:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.320+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7f006c2b0 con 0x7fe7f006c770 2024-03-24T10:33:51.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.321+0000 7fe7ecff9640 1 -- 172.21.15.1:0/379059427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3846 (secure 0 0 0) 0x7fe7e4058590 con 0x7fe7f006c770 2024-03-24T10:33:51.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:51.323 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:33:51.325 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 msgr2=0x7fe7dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 0x7fe7dc06dcd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0007fd0 tx=0x7fe7e0009040 comp rx=0 tx=0).stop 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 msgr2=0x7fe7f0155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400c7f0 tx=0x7fe7e400ccb0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 shutdown_connections 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe7dc06b810 0x7fe7dc06dcd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe7f00b74c0 0x7fe7f015a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7f006d2d0 0x7fe7f015e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 --2- 172.21.15.1:0/379059427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7f006c770 0x7fe7f0155570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 >> 172.21.15.1:0/379059427 conn(0x7fe7f00d1c80 msgr2=0x7fe7f00c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 shutdown_connections 2024-03-24T10:33:51.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.323+0000 7fe7f62d4640 1 -- 172.21.15.1:0/379059427 wait complete. 2024-03-24T10:33:51.336 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:51.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.437+0000 7f84d8607640 1 Processor -- start 2024-03-24T10:33:51.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.437+0000 7f84d8607640 1 -- start start 2024-03-24T10:33:51.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d00b74c0 0x7f84d00d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84d006c770 0x7f84d006cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84d006d100 0x7f84d015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f84d006d5b0 con 0x7f84d006c770 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f84d00d1340 con 0x7f84d006d100 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d8607640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f84d00b4c70 con 0x7f84d00b74c0 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84d006c770 0x7f84d006cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84d006c770 0x7f84d006cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45140/0 (socket says 172.21.15.1:45140) 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 -- 172.21.15.1:0/2392232615 learned_addr learned my addr 172.21.15.1:0/2392232615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d6ba4640 1 --2- 172.21.15.1:0/2392232615 >> v2:172.21.15.3:3300/0 conn(0x7f84d006d100 0x7f84d015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d63a3640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d00b74c0 0x7f84d00d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 -- 172.21.15.1:0/2392232615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d00b74c0 msgr2=0x7f84d00d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d00b74c0 0x7f84d00d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 -- 172.21.15.1:0/2392232615 >> v2:172.21.15.3:3300/0 conn(0x7f84d006d100 msgr2=0x7f84d015d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/2392232615 >> v2:172.21.15.3:3300/0 conn(0x7f84d006d100 0x7f84d015d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.438+0000 7f84d5ba2640 1 -- 172.21.15.1:0/2392232615 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d00b4f50 con 0x7f84d006c770 2024-03-24T10:33:51.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006c770 0x7f84d006cbc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f84d00d1900 tx=0x7f84c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441063160f7232c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d53a1640 1 -- 172.21.15.1:0/2392232615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0031870 con 0x7f84d006c770 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d53a1640 1 -- 172.21.15.1:0/2392232615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c00319d0 con 0x7f84d006c770 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d53a1640 1 -- 172.21.15.1:0/2392232615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0031d60 con 0x7f84d006c770 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 -- 172.21.15.1:0/2392232615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006c770 msgr2=0x7f84d006cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006c770 0x7f84d006cbc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f84d00d1900 tx=0x7f84c002b910 comp rx=0 tx=0).stop 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 -- 172.21.15.1:0/2392232615 shutdown_connections 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 --2- 172.21.15.1:0/2392232615 >> v2:172.21.15.3:3300/0 conn(0x7f84d006d100 0x7f84d015d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006c770 0x7f84d006cbc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 --2- 172.21.15.1:0/2392232615 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d00b74c0 0x7f84d00d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.439+0000 7f84d8607640 1 -- 172.21.15.1:0/2392232615 >> 172.21.15.1:0/2392232615 conn(0x7f84d00d1c80 msgr2=0x7f84d00d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.440+0000 7f84d8607640 1 -- 172.21.15.1:0/2392232615 shutdown_connections 2024-03-24T10:33:51.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.440+0000 7f84d8607640 1 -- 172.21.15.1:0/2392232615 wait complete. 2024-03-24T10:33:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 Processor -- start 2024-03-24T10:33:51.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 -- start start 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d006c770 0x7f84d009c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d00b74c0 0x7f84d009d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f84d014e1e0 con 0x7f84d006d100 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f84d009cb20 con 0x7f84d00b74c0 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.441+0000 7f84d8607640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f84d014d5d0 con 0x7f84d006c770 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45144/0 (socket says 172.21.15.1:45144) 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 -- 172.21.15.1:0/1097113610 learned_addr learned my addr 172.21.15.1:0/1097113610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d63a3640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d006c770 0x7f84d009c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d6ba4640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d00b74c0 0x7f84d009d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 -- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d006c770 msgr2=0x7f84d009c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d006c770 0x7f84d009c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 -- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d00b74c0 msgr2=0x7f84d009d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d00b74c0 0x7f84d009d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 -- 172.21.15.1:0/1097113610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00094d0 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d5ba2640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f84c0000c00 tx=0x7f84c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c003abe0 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c003ad40 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d014d8b0 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0044610 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.442+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d014de70 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.443+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d014e350 con 0x7f84d006d100 2024-03-24T10:33:51.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.444+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f84c004d450 con 0x7f84d006d100 2024-03-24T10:33:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.444+0000 7f84c6ffd640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 0x7f84b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.444+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f84c002d040 con 0x7f84d006d100 2024-03-24T10:33:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.444+0000 7f84d63a3640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 0x7f84b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.444+0000 7f84d63a3640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 0x7f84b406dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.448+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f84c0077170 con 0x7f84d006d100 2024-03-24T10:33:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.632+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84d004f4e0 con 0x7f84d006d100 2024-03-24T10:33:51.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.633+0000 7f84c6ffd640 1 -- 172.21.15.1:0/1097113610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12752 (secure 0 0 0) 0x7f84c007a830 con 0x7f84d006d100 2024-03-24T10:33:51.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:51.634 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:45.015969+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:51.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 msgr2=0x7f84b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 0x7f84b406dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc008040 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 msgr2=0x7f84d014c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f84c0000c00 tx=0x7f84c00046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 shutdown_connections 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f84b406b810 0x7f84b406dcd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d00b74c0 0x7f84d009d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84d006d100 0x7f84d014c7e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 --2- 172.21.15.1:0/1097113610 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84d006c770 0x7f84d009c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 >> 172.21.15.1:0/1097113610 conn(0x7f84d00d1c80 msgr2=0x7f84d006ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 shutdown_connections 2024-03-24T10:33:51.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.635+0000 7f84d8607640 1 -- 172.21.15.1:0/1097113610 wait complete. 2024-03-24T10:33:51.647 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:33:51.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:33:51.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.750+0000 7fceadeeb640 1 Processor -- start 2024-03-24T10:33:51.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.750+0000 7fceadeeb640 1 -- start start 2024-03-24T10:33:51.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fceadeeb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 0x7fcea80d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fceadeeb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 0x7fcea815fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fceadeeb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea815c350 con 0x7fcea815bf70 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fceadeeb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fcea80cdc00 con 0x7fcea815fa40 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fceadeeb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcea8160410 con 0x7fcea80d2780 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea7fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 0x7fcea815fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea6ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 0x7fcea80d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea77fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcea815bf70 0x7fcea80cd4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:36100/0 (socket says 172.21.15.1:36100) 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea7fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 0x7fcea815fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:50638/0 (socket says 172.21.15.1:50638) 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea77fe640 1 -- 172.21.15.1:0/3856310937 learned_addr learned my addr 172.21.15.1:0/3856310937 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea6ffd640 1 -- 172.21.15.1:0/3856310937 >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 msgr2=0x7fcea815fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea6ffd640 1 --2- 172.21.15.1:0/3856310937 >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 0x7fcea815fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea6ffd640 1 -- 172.21.15.1:0/3856310937 >> v1:172.21.15.3:6789/0 conn(0x7fcea815bf70 legacy=0x7fcea80cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.751+0000 7fcea6ffd640 1 -- 172.21.15.1:0/3856310937 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea80cc770 con 0x7fcea80d2780 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.752+0000 7fcea6ffd640 1 --2- 172.21.15.1:0/3856310937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 0x7fcea80d0210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce900097a0 tx=0x7fce9002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1c761d24ab36fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.752+0000 7fcea67fc640 1 -- 172.21.15.1:0/3856310937 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90031830 con 0x7fcea80d2780 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.752+0000 7fcea67fc640 1 -- 172.21.15.1:0/3856310937 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce90031990 con 0x7fcea80d2780 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.752+0000 7fcea67fc640 1 -- 172.21.15.1:0/3856310937 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90031d20 con 0x7fcea80d2780 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 -- 172.21.15.1:0/3856310937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 msgr2=0x7fcea80d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3856310937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 0x7fcea80d0210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce900097a0 tx=0x7fce9002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 -- 172.21.15.1:0/3856310937 shutdown_connections 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3856310937 >> v2:172.21.15.3:3300/0 conn(0x7fcea815fa40 0x7fcea815fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3856310937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea80d2780 0x7fcea80d0210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 -- 172.21.15.1:0/3856310937 >> 172.21.15.1:0/3856310937 conn(0x7fcea80d1c60 msgr2=0x7fcea80d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 -- 172.21.15.1:0/3856310937 shutdown_connections 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.753+0000 7fceadeeb640 1 -- 172.21.15.1:0/3856310937 wait complete. 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.754+0000 7fceadeeb640 1 Processor -- start 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.754+0000 7fceadeeb640 1 -- start start 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea815bf70 0x7fcea80ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcea815fa40 0x7fcea80caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcea815d2f0 con 0x7fcea80d2780 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fcea815c670 con 0x7fcea815fa40 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fceadeeb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcea815c950 con 0x7fcea815bf70 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45160/0 (socket says 172.21.15.1:45160) 2024-03-24T10:33:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 -- 172.21.15.1:0/3130260158 learned_addr learned my addr 172.21.15.1:0/3130260158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea6ffd640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea815bf70 0x7fcea80ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea7fff640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcea815fa40 0x7fcea80caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 -- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea815bf70 msgr2=0x7fcea80ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea815bf70 0x7fcea80ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 -- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcea815fa40 msgr2=0x7fcea80caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcea815fa40 0x7fcea80caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.755+0000 7fcea77fe640 1 -- 172.21.15.1:0/3130260158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce90009400 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fcea77fe640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c400 tx=0x7fce9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98005ce0 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce98005e40 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea815cc30 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce980072d0 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.756+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea815b720 con 0x7fcea80d2780 2024-03-24T10:33:51.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.757+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea8068680 con 0x7fcea80d2780 2024-03-24T10:33:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.757+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fce980074f0 con 0x7fcea80d2780 2024-03-24T10:33:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.757+0000 7fce9ffff640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 0x7fce8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.758+0000 7fcea6ffd640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 0x7fce8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.758+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fce98089360 con 0x7fcea80d2780 2024-03-24T10:33:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.758+0000 7fcea6ffd640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 0x7fce8806dcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce90036000 tx=0x7fce9003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.762+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fce98054210 con 0x7fcea80d2780 2024-03-24T10:33:51.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.941+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcea804f4e0 con 0x7fcea80d2780 2024-03-24T10:33:51.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.942+0000 7fce9ffff640 1 -- 172.21.15.1:0/3130260158 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fce980578d0 con 0x7fcea80d2780 2024-03-24T10:33:51.952 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:33:51.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 msgr2=0x7fce8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 0x7fce8806dcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce90036000 tx=0x7fce9003c040 comp rx=0 tx=0).stop 2024-03-24T10:33:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 msgr2=0x7fcea815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c400 tx=0x7fce9800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 shutdown_connections 2024-03-24T10:33:51.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fce8806b810 0x7fce8806dcd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcea815fa40 0x7fcea80caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcea815bf70 0x7fcea80ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 --2- 172.21.15.1:0/3130260158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea80d2780 0x7fcea815b1e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 >> 172.21.15.1:0/3130260158 conn(0x7fcea80d1c60 msgr2=0x7fcea806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 shutdown_connections 2024-03-24T10:33:51.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:51.944+0000 7fceadeeb640 1 -- 172.21.15.1:0/3130260158 wait complete. 2024-03-24T10:33:51.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T10:33:52.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 Processor -- start 2024-03-24T10:33:52.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 -- start start 2024-03-24T10:33:52.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f401415e190 0x7f40140cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40140d4870 0x7f40141627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4014162d20 con 0x7f40140d2120 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f40140cfdc0 con 0x7f401415e190 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.058+0000 7f4019725640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f40140cf3f0 con 0x7f40140d4870 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f4012ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f401415e190 0x7f40140cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f40137fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f40140d2120 0x7f40140d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39504/0 (socket says 172.21.15.1:39504) 2024-03-24T10:33:52.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f40137fe640 1 -- 172.21.15.1:0/3812094409 learned_addr learned my addr 172.21.15.1:0/3812094409 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f4012ffd640 1 -- 172.21.15.1:0/3812094409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40140d4870 msgr2=0x7f40141627e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f400bfff640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40140d4870 0x7f40141627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f4012ffd640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40140d4870 0x7f40141627e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f4012ffd640 1 -- 172.21.15.1:0/3812094409 >> v1:172.21.15.1:6789/0 conn(0x7f40140d2120 legacy=0x7f40140d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f40127fc640 1 -- 172.21.15.1:0/3812094409 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596221334 0 0) 0x7f400c001490 con 0x7f40140d2120 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.059+0000 7f4012ffd640 1 -- 172.21.15.1:0/3812094409 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40140b3120 con 0x7f401415e190 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4012ffd640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401415e190 0x7f40140cf880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40040097a0 tx=0x7f400402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0bac9af2b49de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f40127fc640 1 -- 172.21.15.1:0/3812094409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004031830 con 0x7f401415e190 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f40127fc640 1 -- 172.21.15.1:0/3812094409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4004031990 con 0x7f401415e190 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f40127fc640 1 -- 172.21.15.1:0/3812094409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004031d20 con 0x7f401415e190 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 -- 172.21.15.1:0/3812094409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401415e190 msgr2=0x7f40140cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401415e190 0x7f40140cf880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40040097a0 tx=0x7f400402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 -- 172.21.15.1:0/3812094409 shutdown_connections 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40140d4870 0x7f40141627e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 --2- 172.21.15.1:0/3812094409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401415e190 0x7f40140cf880 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 -- 172.21.15.1:0/3812094409 >> 172.21.15.1:0/3812094409 conn(0x7f401409d1c0 msgr2=0x7f40140d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 -- 172.21.15.1:0/3812094409 shutdown_connections 2024-03-24T10:33:52.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.060+0000 7f4019725640 1 -- 172.21.15.1:0/3812094409 wait complete. 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.061+0000 7f4019725640 1 Processor -- start 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 -- start start 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 0x7f40140d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 0x7f40141602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f401414d790 con 0x7f40140d2120 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4014157410 con 0x7f40140d4870 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4019725640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f401415e610 con 0x7f401415e190 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4012ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4012ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45184/0 (socket says 172.21.15.1:45184) 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.062+0000 7f4012ffd640 1 -- 172.21.15.1:0/3996665762 learned_addr learned my addr 172.21.15.1:0/3996665762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f400bfff640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 0x7f40140d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f40137fe640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 0x7f40141602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 -- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 msgr2=0x7f40141602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 0x7f40141602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 -- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 msgr2=0x7f40140d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 0x7f40140d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 -- 172.21.15.1:0/3996665762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4004009400 con 0x7f40140d2120 2024-03-24T10:33:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f40137fe640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 0x7f40141602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f400bfff640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 0x7f40140d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4012ffd640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4004034040 tx=0x7f40040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400403aa60 con 0x7f40140d2120 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f400403abe0 con 0x7f40140d2120 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401415e8b0 con 0x7f40140d2120 2024-03-24T10:33:52.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004044400 con 0x7f40140d2120 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.063+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401415ee70 con 0x7f40140d2120 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.064+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4014161820 con 0x7f40140d2120 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.064+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f400404d440 con 0x7f40140d2120 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.065+0000 7f400b7fe640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 0x7f3ff406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.065+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f40040ac190 con 0x7f40140d2120 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.065+0000 7f400bfff640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 0x7f3ff406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.065+0000 7f400bfff640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 0x7f3ff406dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4014161f10 tx=0x7f3ffc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.069+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4004076e70 con 0x7f40140d2120 2024-03-24T10:33:52.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.253+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f401404f4e0 con 0x7f40140d2120 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.253+0000 7f400b7fe640 1 -- 172.21.15.1:0/3996665762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f4004035090 con 0x7f40140d2120 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-24T10:33:52.255 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:33:52.256 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.255+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 msgr2=0x7f3ff406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.255+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 0x7f3ff406dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4014161f10 tx=0x7f3ffc008040 comp rx=0 tx=0).stop 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.255+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 msgr2=0x7f401415f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.255+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4004034040 tx=0x7f40040046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 shutdown_connections 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ff406b810 0x7f3ff406dcd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f401415e190 0x7f40141602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40140d4870 0x7f40140d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 --2- 172.21.15.1:0/3996665762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40140d2120 0x7f401415f390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 >> 172.21.15.1:0/3996665762 conn(0x7f401409d1c0 msgr2=0x7f40140b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 shutdown_connections 2024-03-24T10:33:52.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.256+0000 7f4019725640 1 -- 172.21.15.1:0/3996665762 wait complete. 2024-03-24T10:33:52.267 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T10:33:52.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T10:33:52.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.371+0000 7ff0e8725640 1 Processor -- start 2024-03-24T10:33:52.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.372+0000 7ff0e8725640 1 -- start start 2024-03-24T10:33:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.372+0000 7ff0e8725640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.372+0000 7ff0e8725640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e00cc250 con 0x7ff0e006d070 2024-03-24T10:33:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.372+0000 7ff0e8725640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e00d1060 con 0x7ff0e00b31d0 2024-03-24T10:33:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.372+0000 7ff0e8725640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e00b35b0 con 0x7ff0e00b1380 2024-03-24T10:33:52.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e6cc2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff0e006d070 0x7ff0e00ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39516/0 (socket says 172.21.15.1:39516) 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e6cc2640 1 -- 172.21.15.1:0/2417606043 learned_addr learned my addr 172.21.15.1:0/2417606043 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e5cc0640 1 --2- 172.21.15.1:0/2417606043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454957880 0 0) 0x7ff0e00cc250 con 0x7ff0e006d070 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0cc003520 con 0x7ff0e006d070 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029272680 0 0) 0x7ff0e00d1060 con 0x7ff0e00b31d0 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e00cc250 con 0x7ff0e00b31d0 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e5cc0640 1 -- 172.21.15.1:0/2417606043 >> v1:172.21.15.3:6789/0 conn(0x7ff0e00b31d0 legacy=0x7ff0e00d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e5cc0640 1 -- 172.21.15.1:0/2417606043 >> v1:172.21.15.1:6789/0 conn(0x7ff0e006d070 legacy=0x7ff0e00ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3427908933 0 0) 0x7ff0cc003520 con 0x7ff0e006d070 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.373+0000 7ff0e5cc0640 1 -- 172.21.15.1:0/2417606043 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e00cc4a0 con 0x7ff0e00b1380 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.374+0000 7ff0e5cc0640 1 --2- 172.21.15.1:0/2417606043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e006cb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097d0 tx=0x7ff0d002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bf1d2fb32fc3fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.374+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80019f0 con 0x7ff0e00b1380 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.374+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0d002e050 con 0x7ff0e00b1380 2024-03-24T10:33:52.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.374+0000 7ff0e54bf640 1 -- 172.21.15.1:0/2417606043 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0030730 con 0x7ff0e00b1380 2024-03-24T10:33:52.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 -- 172.21.15.1:0/2417606043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 msgr2=0x7ff0e006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2417606043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e006cb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097d0 tx=0x7ff0d002b800 comp rx=0 tx=0).stop 2024-03-24T10:33:52.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 -- 172.21.15.1:0/2417606043 shutdown_connections 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2417606043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e006cb30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 -- 172.21.15.1:0/2417606043 >> 172.21.15.1:0/2417606043 conn(0x7ff0e00d1c00 msgr2=0x7ff0e00d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 -- 172.21.15.1:0/2417606043 shutdown_connections 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.375+0000 7ff0e8725640 1 -- 172.21.15.1:0/2417606043 wait complete. 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.376+0000 7ff0e8725640 1 Processor -- start 2024-03-24T10:33:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.376+0000 7ff0e8725640 1 -- start start 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e006d070 0x7ff0e00c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e00c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e006d520 con 0x7ff0e00b31d0 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e00cbb40 con 0x7ff0e006d070 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e00c6a20 con 0x7ff0e00b1380 2024-03-24T10:33:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45186/0 (socket says 172.21.15.1:45186) 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 -- 172.21.15.1:0/2183629616 learned_addr learned my addr 172.21.15.1:0/2183629616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e5cc0640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e00c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e64c1640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e006d070 0x7ff0e00c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 -- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 msgr2=0x7ff0e00c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e00c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 -- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e006d070 msgr2=0x7ff0e00c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e006d070 0x7ff0e00c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 -- 172.21.15.1:0/2183629616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d0009400 con 0x7ff0e00b31d0 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e6cc2640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800cc50 tx=0x7ff0d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800ce60 con 0x7ff0e00b31d0 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0d8012400 con 0x7ff0e00b31d0 2024-03-24T10:33:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e00c5e10 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8012790 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.377+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e00c63d0 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.378+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e004f470 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.379+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff0d80128f0 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.379+0000 7ff0d6ffd640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 0x7ff0c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.379+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff0d808a4a0 con 0x7ff0e00b31d0 2024-03-24T10:33:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.379+0000 7ff0e64c1640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 0x7ff0c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.380+0000 7ff0e64c1640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 0x7ff0c406dd00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0011e0 tx=0x7ff0dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.383+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff0d80552c0 con 0x7ff0e00b31d0 2024-03-24T10:33:52.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:52.563+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff0e00c6b60 con 0x7ff0e00b31d0 2024-03-24T10:33:53.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.102+0000 7ff0d6ffd640 1 -- 172.21.15.1:0/2183629616 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7ff0d8058980 con 0x7ff0e00b31d0 2024-03-24T10:33:53.104 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-24T10:33:53.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 msgr2=0x7ff0c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 0x7ff0c406dd00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0011e0 tx=0x7ff0dc009040 comp rx=0 tx=0).stop 2024-03-24T10:33:53.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 msgr2=0x7ff0e01530b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800cc50 tx=0x7ff0d800a430 comp rx=0 tx=0).stop 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 shutdown_connections 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff0c406b840 0x7ff0c406dd00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e00b31d0 0x7ff0e01530b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0e00b1380 0x7ff0e00c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 --2- 172.21.15.1:0/2183629616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0e006d070 0x7ff0e00c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.106+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 >> 172.21.15.1:0/2183629616 conn(0x7ff0e00d1c00 msgr2=0x7ff0e0074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:53.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.107+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 shutdown_connections 2024-03-24T10:33:53.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.107+0000 7ff0e8725640 1 -- 172.21.15.1:0/2183629616 wait complete. 2024-03-24T10:33:53.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T10:33:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.244+0000 7f14bcc3b640 1 Processor -- start 2024-03-24T10:33:53.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.244+0000 7f14bcc3b640 1 -- start start 2024-03-24T10:33:53.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14bcc3b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14b80d2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14bcc3b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f14b80b7a00 0x7f14b80b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14bcc3b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b80b7570 con 0x7f14b80d0d00 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14bcc3b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f14b80b6f60 con 0x7f14b80b7a00 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14bcc3b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f14b80d2a90 con 0x7f14b80d0920 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14affff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14b80d0d00 0x7f14b80d1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39532/0 (socket says 172.21.15.1:39532) 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14affff640 1 -- 172.21.15.1:0/1264791626 learned_addr learned my addr 172.21.15.1:0/1264791626 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b659c640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14b80d2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b6d9d640 1 --2- 172.21.15.1:0/1264791626 >> v2:172.21.15.3:3300/0 conn(0x7f14b80b7a00 0x7f14b80b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 398577659 0 0) 0x7f14b80b7570 con 0x7f14b80d0d00 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14a4003520 con 0x7f14b80d0d00 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1248525111 0 0) 0x7f14a4003520 con 0x7f14b80d0d00 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 msgr2=0x7f14b80d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14b80d2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 >> v2:172.21.15.3:3300/0 conn(0x7f14b80b7a00 msgr2=0x7f14b80b7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:53.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 --2- 172.21.15.1:0/1264791626 >> v2:172.21.15.3:3300/0 conn(0x7f14b80b7a00 0x7f14b80b7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b80d3010 con 0x7f14b80d0d00 2024-03-24T10:33:53.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f14a0001670 con 0x7f14b80d0d00 2024-03-24T10:33:53.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 >> v1:172.21.15.1:6789/0 conn(0x7f14b80d0d00 legacy=0x7f14b80d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14a4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.245+0000 7f14b5d9b640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14a4003ef0 0x7f14a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b5d9b640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a4004d00 0x7f14a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f14b80b6f60 con 0x7f14a4003ef0 2024-03-24T10:33:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f14b80d2a90 con 0x7f14a4004d00 2024-03-24T10:33:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f14a400b980 con 0x7f14b80d0920 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14a4003ef0 0x7f14a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b659c640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14a4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b6d9d640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a4004d00 0x7f14a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 -- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 msgr2=0x7f14a4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14b80d0920 0x7f14a4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 -- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a4004d00 msgr2=0x7f14a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a4004d00 0x7f14a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14affff640 1 -- 172.21.15.1:0/1264791626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a4003520 con 0x7f14a4003ef0 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.246+0000 7f14b6d9d640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a4004d00 0x7f14a4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14affff640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14a4003ef0 0x7f14a40047c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c240 tx=0x7f14a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add6a287153b601 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a002cae0 con 0x7f14a4003ef0 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a002cc40 con 0x7f14a4003ef0 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14b5d9b640 1 -- 172.21.15.1:0/1264791626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0005b10 con 0x7f14a4003ef0 2024-03-24T10:33:53.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 -- 172.21.15.1:0/1264791626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14a4003ef0 msgr2=0x7f14a40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/1264791626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14a4003ef0 0x7f14a40047c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c240 tx=0x7f14a40057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14b6d9d640 1 -- 172.21.15.1:0/1264791626 reap_dead start 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 -- 172.21.15.1:0/1264791626 shutdown_connections 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 -- 172.21.15.1:0/1264791626 >> 172.21.15.1:0/1264791626 conn(0x7f14b80d3f70 msgr2=0x7f14b80d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 -- 172.21.15.1:0/1264791626 shutdown_connections 2024-03-24T10:33:53.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.247+0000 7f14bcc3b640 1 -- 172.21.15.1:0/1264791626 wait complete. 2024-03-24T10:33:53.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 Processor -- start 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 -- start start 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14a4003ef0 0x7f14b80c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 0x7f14b81609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f14b815ba50 con 0x7f14b815b5d0 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f14b815f3c0 con 0x7f14b8152680 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14bcc3b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f14b815f5f0 con 0x7f14a4003ef0 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45208/0 (socket says 172.21.15.1:45208) 2024-03-24T10:33:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.249+0000 7f14affff640 1 -- 172.21.15.1:0/3210134699 learned_addr learned my addr 172.21.15.1:0/3210134699 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:53.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14b659c640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14a4003ef0 0x7f14b80c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14b6d9d640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 0x7f14b81609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 -- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14a4003ef0 msgr2=0x7f14b80c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14a4003ef0 0x7f14b80c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 -- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 msgr2=0x7f14b81609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 0x7f14b81609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 -- 172.21.15.1:0/3210134699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b815f8d0 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14b6d9d640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 0x7f14b81609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14affff640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14a002c5b0 tx=0x7f14a0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0004bf0 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a0004d50 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b815fbb0 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0007270 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.250+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b815fe90 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.251+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b804f470 con 0x7f14b815b5d0 2024-03-24T10:33:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.252+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f14a0032070 con 0x7f14b815b5d0 2024-03-24T10:33:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.252+0000 7f14aeffd640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 0x7f149806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.252+0000 7f14b659c640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 0x7f149806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.252+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f14a00aad30 con 0x7f14b815b5d0 2024-03-24T10:33:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.252+0000 7f14b659c640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 0x7f149806dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009ae0 tx=0x7f14a8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.257+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f14a0075b50 con 0x7f14b815b5d0 2024-03-24T10:33:53.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:53.437+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f14b815eac0 con 0x7f14b815b5d0 2024-03-24T10:33:54.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.105+0000 7f14aeffd640 1 -- 172.21.15.1:0/3210134699 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f14a0045020 con 0x7f14b815b5d0 2024-03-24T10:33:54.106 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 msgr2=0x7f149806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 0x7f149806dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009ae0 tx=0x7f14a8009190 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 msgr2=0x7f14b815ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14a002c5b0 tx=0x7f14a0001a00 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 shutdown_connections 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f149806b840 0x7f149806dd00 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009ae0 tx=0x7f14a8009190 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14b8152680 0x7f14b81609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14b815b5d0 0x7f14b815ee80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 --2- 172.21.15.1:0/3210134699 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f14a4003ef0 0x7f14b80c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.110+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 >> 172.21.15.1:0/3210134699 conn(0x7f14b80d3f70 msgr2=0x7f14b815d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:54.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.111+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 shutdown_connections 2024-03-24T10:33:54.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.111+0000 7f14bcc3b640 1 -- 172.21.15.1:0/3210134699 wait complete. 2024-03-24T10:33:54.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 Processor -- start 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 -- start start 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb2c160030 0x7fbb2c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2c160630 con 0x7fbb2c144a10 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2c06cf80 con 0x7fbb2c0cff00 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb30997640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbb2c06e170 con 0x7fbb2c160030 2024-03-24T10:33:54.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.248+0000 7fbb2a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb2c144a10 0x7fbb2c06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39540/0 (socket says 172.21.15.1:39540) 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 -- 172.21.15.1:0/1270891311 learned_addr learned my addr 172.21.15.1:0/1270891311 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/1270891311 >> v2:172.21.15.3:3300/0 conn(0x7fbb2c160030 0x7fbb2c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704592515 0 0) 0x7fbb2c160630 con 0x7fbb2c144a10 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb18003520 con 0x7fbb2c144a10 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 244735751 0 0) 0x7fbb18003520 con 0x7fbb2c144a10 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 >> v2:172.21.15.3:3300/0 conn(0x7fbb2c160030 msgr2=0x7fbb2c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 --2- 172.21.15.1:0/1270891311 >> v2:172.21.15.3:3300/0 conn(0x7fbb2c160030 0x7fbb2c0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 >> v1:172.21.15.3:6789/0 conn(0x7fbb2c0cff00 legacy=0x7fbb2c15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c06d260 con 0x7fbb2c144a10 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fbb1c001670 con 0x7fbb2c144a10 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 >> v1:172.21.15.1:6789/0 conn(0x7fbb2c144a10 legacy=0x7fbb2c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 0x7fbb18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb180047c0 0x7fbb18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18005180 0x7fbb1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c06e170 con 0x7fbb18003970 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1800c020 con 0x7fbb180047c0 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1800c1e0 con 0x7fbb18005180 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 0x7fbb18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb23fff640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb180047c0 0x7fbb18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18005180 0x7fbb1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 -- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18005180 msgr2=0x7fbb1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18005180 0x7fbb1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 -- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb180047c0 msgr2=0x7fbb18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb180047c0 0x7fbb18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.249+0000 7fbb2a59c640 1 -- 172.21.15.1:0/1270891311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb18003520 con 0x7fbb18003970 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb23fff640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb180047c0 0x7fbb18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18005180 0x7fbb1800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb2ad9d640 1 -- 172.21.15.1:0/1270891311 reap_dead start 2024-03-24T10:33:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 0x7fbb18003d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c001050 tx=0x7fbb180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1294d1c7865e0ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c005ad0 con 0x7fbb18003970 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1c005c30 con 0x7fbb18003970 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.250+0000 7fbb29d9b640 1 -- 172.21.15.1:0/1270891311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c02ca50 con 0x7fbb18003970 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 -- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 msgr2=0x7fbb18003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 0x7fbb18003d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c001050 tx=0x7fbb180057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 -- 172.21.15.1:0/1270891311 shutdown_connections 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 --2- 172.21.15.1:0/1270891311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb18003970 0x7fbb18003d90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 -- 172.21.15.1:0/1270891311 >> 172.21.15.1:0/1270891311 conn(0x7fbb2c06b4f0 msgr2=0x7fbb2c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 -- 172.21.15.1:0/1270891311 shutdown_connections 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.251+0000 7fbb30997640 1 -- 172.21.15.1:0/1270891311 wait complete. 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.252+0000 7fbb30997640 1 Processor -- start 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.252+0000 7fbb30997640 1 -- start start 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 0x7fbb2c15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 0x7fbb2c139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c143d40 con 0x7fbb2c143380 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1e6820 con 0x7fbb2c1e6030 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb30997640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb2c1e6b00 con 0x7fbb18003970 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45234/0 (socket says 172.21.15.1:45234) 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 -- 172.21.15.1:0/4224035825 learned_addr learned my addr 172.21.15.1:0/4224035825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb23fff640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 0x7fbb2c139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 0x7fbb2c15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 -- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 msgr2=0x7fbb2c15e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 0x7fbb2c15e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 -- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 msgr2=0x7fbb2c139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 0x7fbb2c139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2ad9d640 1 -- 172.21.15.1:0/4224035825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c144f70 con 0x7fbb2c143380 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb23fff640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 0x7fbb2c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.253+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 0x7fbb2c15e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb2ad9d640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0b5200 tx=0x7fbb24010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2401b070 con 0x7fbb2c143380 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2400f040 con 0x7fbb2c143380 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2c1e6de0 con 0x7fbb2c143380 2024-03-24T10:33:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400e050 con 0x7fbb2c143380 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.254+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2c1e73a0 con 0x7fbb2c143380 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.255+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2c04f470 con 0x7fbb2c143380 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.255+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb24015440 con 0x7fbb2c143380 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.256+0000 7fbb22ffd640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 0x7fbb0c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.256+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fbb2408a480 con 0x7fbb2c143380 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.256+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 0x7fbb0c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.256+0000 7fbb2a59c640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 0x7fbb0c06dd00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c034000 tx=0x7fbb1c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:54.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.260+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb24054f90 con 0x7fbb2c143380 2024-03-24T10:33:54.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:54.480+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbb2c072ad0 con 0x7fbb2c143380 2024-03-24T10:33:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.130+0000 7fbb22ffd640 1 -- 172.21.15.1:0/4224035825 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fbb24058650 con 0x7fbb2c143380 2024-03-24T10:33:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T10:33:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 msgr2=0x7fbb0c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 0x7fbb0c06dd00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c034000 tx=0x7fbb1c007000 comp rx=0 tx=0).stop 2024-03-24T10:33:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 msgr2=0x7fbb2c143800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0b5200 tx=0x7fbb24010590 comp rx=0 tx=0).stop 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 shutdown_connections 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb0c06b840 0x7fbb0c06dd00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb2c143380 0x7fbb2c143800 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c1e6030 0x7fbb2c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 --2- 172.21.15.1:0/4224035825 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb18003970 0x7fbb2c15e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.137+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 >> 172.21.15.1:0/4224035825 conn(0x7fbb2c06b4f0 msgr2=0x7fbb2c0b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.139+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 shutdown_connections 2024-03-24T10:33:55.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.139+0000 7fbb30997640 1 -- 172.21.15.1:0/4224035825 wait complete. 2024-03-24T10:33:55.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.267+0000 7fee5b7b6640 1 Processor -- start 2024-03-24T10:33:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.267+0000 7fee5b7b6640 1 -- start start 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee5b7b6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee54166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee5b7b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5406ce40 con 0x7fee5406c910 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee5b7b6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5406d120 con 0x7fee5415bda0 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee5b7b6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fee540b2db0 con 0x7fee540d00d0 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee58d51640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fee5406c910 0x7fee540cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39548/0 (socket says 172.21.15.1:39548) 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee58d51640 1 -- 172.21.15.1:0/1332812119 learned_addr learned my addr 172.21.15.1:0/1332812119 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576505559 0 0) 0x7fee5406ce40 con 0x7fee5406c910 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee3c003520 con 0x7fee5406c910 2024-03-24T10:33:55.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.268+0000 7fee59d53640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee54166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 943900195 0 0) 0x7fee3c003520 con 0x7fee5406c910 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 msgr2=0x7fee54166a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee54166a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 >> v1:172.21.15.3:6789/0 conn(0x7fee5415bda0 legacy=0x7fee5415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5406d400 con 0x7fee5406c910 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fee48001670 con 0x7fee5406c910 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59d53640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee54166a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 >> v1:172.21.15.1:6789/0 conn(0x7fee5406c910 legacy=0x7fee540cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee3c0039a0 0x7fee3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee3c004d00 0x7fee3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c00bb10 con 0x7fee3c0039a0 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c00bc80 con 0x7fee3c004d00 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fee3c00be40 con 0x7fee540d00d0 2024-03-24T10:33:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee3c0039a0 0x7fee3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee58d51640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59d53640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee3c004d00 0x7fee3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 -- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 msgr2=0x7fee3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee540d00d0 0x7fee3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 -- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee3c004d00 msgr2=0x7fee3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee3c004d00 0x7fee3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.269+0000 7fee59552640 1 -- 172.21.15.1:0/1332812119 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c003520 con 0x7fee3c0039a0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.270+0000 7fee59552640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee3c0039a0 0x7fee3c003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001750 tx=0x7fee3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d6e2bca4249cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.270+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0052d0 con 0x7fee3c0039a0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.270+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee4c02e050 con 0x7fee3c0039a0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.270+0000 7fee53fff640 1 -- 172.21.15.1:0/1332812119 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0035f0 con 0x7fee3c0039a0 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 -- 172.21.15.1:0/1332812119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee3c0039a0 msgr2=0x7fee3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/1332812119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee3c0039a0 0x7fee3c003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001750 tx=0x7fee3c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee59d53640 1 -- 172.21.15.1:0/1332812119 reap_dead start 2024-03-24T10:33:55.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 -- 172.21.15.1:0/1332812119 shutdown_connections 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 -- 172.21.15.1:0/1332812119 >> 172.21.15.1:0/1332812119 conn(0x7fee540b17a0 msgr2=0x7fee540b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 -- 172.21.15.1:0/1332812119 shutdown_connections 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.271+0000 7fee5b7b6640 1 -- 172.21.15.1:0/1332812119 wait complete. 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.272+0000 7fee5b7b6640 1 Processor -- start 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.272+0000 7fee5b7b6640 1 -- start start 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 0x7fee54147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee5414a720 0x7fee541527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fee54144f80 con 0x7fee54146d70 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fee54147730 con 0x7fee5414a720 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fee5414ab80 con 0x7fee3c0039a0 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45256/0 (socket says 172.21.15.1:45256) 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 -- 172.21.15.1:0/2072865353 learned_addr learned my addr 172.21.15.1:0/2072865353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59552640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 0x7fee54147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee58d51640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee5414a720 0x7fee541527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 -- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 msgr2=0x7fee54147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 0x7fee54147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 -- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee5414a720 msgr2=0x7fee541527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee5414a720 0x7fee541527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 -- 172.21.15.1:0/2072865353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee54153010 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59552640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 0x7fee54147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee59d53640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee540cd520 tx=0x7fee44010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4401b070 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee44010d20 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee541532f0 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.273+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44010ed0 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.274+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee54152cf0 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.274+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee54153810 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.275+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee44003990 con 0x7fee54146d70 2024-03-24T10:33:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.275+0000 7fee51ffb640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 0x7fee4006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.275+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fee4408a930 con 0x7fee54146d70 2024-03-24T10:33:55.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.275+0000 7fee59552640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 0x7fee4006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.276+0000 7fee59552640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 0x7fee4006dcd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee4c033000 tx=0x7fee4c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.280+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee54153810 con 0x7fee54146d70 2024-03-24T10:33:55.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.459+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee54153810 con 0x7fee54146d70 2024-03-24T10:33:55.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.461+0000 7fee51ffb640 1 -- 172.21.15.1:0/2072865353 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16486 (secure 0 0 0) 0x7fee54153810 con 0x7fee54146d70 2024-03-24T10:33:55.462 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:55.462 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:54.482897+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 msgr2=0x7fee4006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 0x7fee4006dcd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee4c033000 tx=0x7fee4c009030 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 msgr2=0x7fee541471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee540cd520 tx=0x7fee44010530 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 shutdown_connections 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee4006b810 0x7fee4006dcd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee54146d70 0x7fee541471f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee5414a720 0x7fee541527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 --2- 172.21.15.1:0/2072865353 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee3c0039a0 0x7fee54147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 >> 172.21.15.1:0/2072865353 conn(0x7fee540b17a0 msgr2=0x7fee540cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 shutdown_connections 2024-03-24T10:33:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.463+0000 7fee5b7b6640 1 -- 172.21.15.1:0/2072865353 wait complete. 2024-03-24T10:33:55.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:55.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.589+0000 7f2f3129a640 1 Processor -- start 2024-03-24T10:33:55.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.589+0000 7f2f3129a640 1 -- start start 2024-03-24T10:33:55.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f3129a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f2c0ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f3129a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2c0cf1a0 con 0x7f2f2c15e150 2024-03-24T10:33:55.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f3129a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2c0cf480 con 0x7f2f2c0b51f0 2024-03-24T10:33:55.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f3129a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c0cf760 con 0x7f2f2c15e530 2024-03-24T10:33:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f2affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f2c15e150 0x7f2f2c0b4ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39564/0 (socket says 172.21.15.1:39564) 2024-03-24T10:33:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f2affd640 1 -- 172.21.15.1:0/2845028053 learned_addr learned my addr 172.21.15.1:0/2845028053 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.590+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f2c0ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457428586 0 0) 0x7f2f2c0cf1a0 con 0x7f2f2c15e150 2024-03-24T10:33:55.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f18003520 con 0x7f2f2c15e150 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645014289 0 0) 0x7f2f2c0cf480 con 0x7f2f2c0b51f0 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2c0cf1a0 con 0x7f2f2c0b51f0 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2433379203 0 0) 0x7f2f18003520 con 0x7f2f2c15e150 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 msgr2=0x7f2f2c0ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f2c0ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 >> v1:172.21.15.3:6789/0 conn(0x7f2f2c0b51f0 legacy=0x7f2f2c0d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c0cfa40 con 0x7f2f2c15e150 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f2f1c0019f0 con 0x7f2f2c15e150 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 >> v1:172.21.15.1:6789/0 conn(0x7f2f2c15e150 legacy=0x7f2f2c0b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f18003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f18004d00 0x7f2f18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0cf760 con 0x7f2f18004090 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1800ba60 con 0x7f2f18004d00 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1800bc20 con 0x7f2f2c15e530 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2affd640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f18003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2b7fe640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f18004d00 0x7f2f18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 -- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 msgr2=0x7f2f18003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c15e530 0x7f2f18003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 -- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f18004d00 msgr2=0x7f2f18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f18004d00 0x7f2f18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.591+0000 7f2f2a7fc640 1 -- 172.21.15.1:0/2845028053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f18003520 con 0x7f2f18004090 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.592+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f180047c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009ac0 tx=0x7f2f180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bc68787def3599 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.592+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1402c6b0 con 0x7f2f18004090 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.592+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f1402c810 con 0x7f2f18004090 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.592+0000 7f2f29ffb640 1 -- 172.21.15.1:0/2845028053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1402cba0 con 0x7f2f18004090 2024-03-24T10:33:55.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 -- 172.21.15.1:0/2845028053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 msgr2=0x7f2f180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2845028053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f180047c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009ac0 tx=0x7f2f180057f0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f2b7fe640 1 -- 172.21.15.1:0/2845028053 reap_dead start 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 -- 172.21.15.1:0/2845028053 shutdown_connections 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 -- 172.21.15.1:0/2845028053 >> 172.21.15.1:0/2845028053 conn(0x7f2f2c09d1b0 msgr2=0x7f2f2c0d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 -- 172.21.15.1:0/2845028053 shutdown_connections 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.593+0000 7f2f3129a640 1 -- 172.21.15.1:0/2845028053 wait complete. 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.594+0000 7f2f3129a640 1 Processor -- start 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.594+0000 7f2f3129a640 1 -- start start 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c0ca790 0x7f2f2c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f2c0cabf0 0x7f2f2c0d2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0c96f0 con 0x7f2f18004090 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0c99d0 con 0x7f2f2c0cabf0 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f3129a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c0c9cb0 con 0x7f2f2c0ca790 2024-03-24T10:33:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45274/0 (socket says 172.21.15.1:45274) 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 -- 172.21.15.1:0/2426146346 learned_addr learned my addr 172.21.15.1:0/2426146346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2b7fe640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f2c0cabf0 0x7f2f2c0d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c0ca790 0x7f2f2c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 -- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c0ca790 msgr2=0x7f2f2c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c0ca790 0x7f2f2c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 -- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f2c0cabf0 msgr2=0x7f2f2c0d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f2c0cabf0 0x7f2f2c0d2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 -- 172.21.15.1:0/2426146346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c0c9f90 con 0x7f2f18004090 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.595+0000 7f2f2affd640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0079f0 tx=0x7f2f1c00a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.596+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c007ae0 con 0x7f2f18004090 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.596+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f1c0133f0 con 0x7f2f18004090 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.596+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c0ca270 con 0x7f2f18004090 2024-03-24T10:33:55.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.596+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0137c0 con 0x7f2f18004090 2024-03-24T10:33:55.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.596+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c0b0aa0 con 0x7f2f18004090 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.597+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c04f4e0 con 0x7f2f18004090 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.597+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f1c013920 con 0x7f2f18004090 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.597+0000 7f2f237fe640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 0x7f2f0c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.597+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2f1c08add0 con 0x7f2f18004090 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.598+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 0x7f2f0c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.598+0000 7f2f2a7fc640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 0x7f2f0c06dd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f14036000 tx=0x7f2f14030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.602+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f1c055930 con 0x7f2f18004090 2024-03-24T10:33:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.787+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f2c0b6210 con 0x7f2f18004090 2024-03-24T10:33:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.788+0000 7f2f237fe640 1 -- 172.21.15.1:0/2426146346 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16486 (secure 0 0 0) 0x7f2f1c08c090 con 0x7f2f18004090 2024-03-24T10:33:55.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:55.796 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:54.482897+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 msgr2=0x7f2f0c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 0x7f2f0c06dd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f14036000 tx=0x7f2f14030000 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 msgr2=0x7f2f2c15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0079f0 tx=0x7f2f1c00a6a0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 shutdown_connections 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b840 0x7f2f0c06dd00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f2c0cabf0 0x7f2f2c0d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f2c0ca790 0x7f2f2c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 --2- 172.21.15.1:0/2426146346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f18004090 0x7f2f2c15b3c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 >> 172.21.15.1:0/2426146346 conn(0x7f2f2c09d1b0 msgr2=0x7f2f2c15d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 shutdown_connections 2024-03-24T10:33:55.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.790+0000 7f2f3129a640 1 -- 172.21.15.1:0/2426146346 wait complete. 2024-03-24T10:33:55.803 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T10:33:55.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:33:55.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:33:55.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.906+0000 7f917202a640 1 Processor -- start 2024-03-24T10:33:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.906+0000 7f917202a640 1 -- start start 2024-03-24T10:33:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f917202a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c0b4e40 0x7f916c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f917202a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f916c162610 0x7f916c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f917202a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f916c0c7a40 con 0x7f916c1615f0 2024-03-24T10:33:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f917202a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f916c0cfd50 con 0x7f916c162610 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f917202a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f916c0b55b0 con 0x7f916c0b4e40 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f916c162610 0x7f916c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f916c162610 0x7f916c0d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45286/0 (socket says 172.21.15.1:45286) 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 -- 172.21.15.1:0/3610676072 learned_addr learned my addr 172.21.15.1:0/3610676072 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 -- 172.21.15.1:0/3610676072 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c0b4e40 msgr2=0x7f916c0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 --2- 172.21.15.1:0/3610676072 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c0b4e40 0x7f916c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 -- 172.21.15.1:0/3610676072 >> v1:172.21.15.3:6789/0 conn(0x7f916c1615f0 legacy=0x7f916c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.907+0000 7f916bfff640 1 -- 172.21.15.1:0/3610676072 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c0b52d0 con 0x7f916c162610 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.908+0000 7f916bfff640 1 --2- 172.21.15.1:0/3610676072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c0d1c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9164009e50 tx=0x7f916402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53a2fc109454e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.908+0000 7f916a7fc640 1 -- 172.21.15.1:0/3610676072 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916402ca70 con 0x7f916c162610 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.908+0000 7f916a7fc640 1 -- 172.21.15.1:0/3610676072 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916402cbd0 con 0x7f916c162610 2024-03-24T10:33:55.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.908+0000 7f916a7fc640 1 -- 172.21.15.1:0/3610676072 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640343f0 con 0x7f916c162610 2024-03-24T10:33:55.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 -- 172.21.15.1:0/3610676072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 msgr2=0x7f916c0d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 --2- 172.21.15.1:0/3610676072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c0d1c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9164009e50 tx=0x7f916402bed0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 -- 172.21.15.1:0/3610676072 shutdown_connections 2024-03-24T10:33:55.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 --2- 172.21.15.1:0/3610676072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c0d1c70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 --2- 172.21.15.1:0/3610676072 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c0b4e40 0x7f916c0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 -- 172.21.15.1:0/3610676072 >> 172.21.15.1:0/3610676072 conn(0x7f916c0d3ef0 msgr2=0x7f916c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 -- 172.21.15.1:0/3610676072 shutdown_connections 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.909+0000 7f917202a640 1 -- 172.21.15.1:0/3610676072 wait complete. 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.910+0000 7f917202a640 1 Processor -- start 2024-03-24T10:33:55.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.910+0000 7f917202a640 1 -- start start 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c1615f0 0x7f916c1f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f916c1532f0 0x7f916c15f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1f8210 con 0x7f916c162610 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f916c15ee80 con 0x7f916c1532f0 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f917202a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f916c1605e0 con 0x7f916c1615f0 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45302/0 (socket says 172.21.15.1:45302) 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 -- 172.21.15.1:0/2511275481 learned_addr learned my addr 172.21.15.1:0/2511275481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916b7fe640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c1615f0 0x7f916c1f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916bfff640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f916c1532f0 0x7f916c15f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 -- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c1615f0 msgr2=0x7f916c1f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c1615f0 0x7f916c1f72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 -- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f916c1532f0 msgr2=0x7f916c15f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f916c1532f0 0x7f916c15f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 -- 172.21.15.1:0/2511275481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9164009b30 con 0x7f916c162610 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f916affd640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f916000b4a0 tx=0x7f916000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.911+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160010610 con 0x7f916c162610 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.912+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916c1f7970 con 0x7f916c162610 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.912+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9160010770 con 0x7f916c162610 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.912+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160010b20 con 0x7f916c162610 2024-03-24T10:33:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.912+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916c1f7f30 con 0x7f916c162610 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.912+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f916c15efc0 con 0x7f916c162610 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.913+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9160010c80 con 0x7f916c162610 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.913+0000 7f9153fff640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 0x7f915806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.913+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9160089e50 con 0x7f916c162610 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.913+0000 7f916b7fe640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 0x7f915806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.914+0000 7f916b7fe640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 0x7f915806dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f915c007fd0 tx=0x7f915c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:55.917+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f916c15efc0 con 0x7f916c162610 2024-03-24T10:33:56.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:56.138+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f916c15efc0 con 0x7f916c162610 2024-03-24T10:33:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.137+0000 7f9153fff640 1 -- 172.21.15.1:0/2511275481 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f916c15efc0 con 0x7f916c162610 2024-03-24T10:33:57.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 msgr2=0x7f915806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 0x7f915806dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f915c007fd0 tx=0x7f915c009040 comp rx=0 tx=0).stop 2024-03-24T10:33:57.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 msgr2=0x7f916c1600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f916000b4a0 tx=0x7f916000b960 comp rx=0 tx=0).stop 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 shutdown_connections 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f915806b810 0x7f915806dcd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f916c1532f0 0x7f916c15f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f916c162610 0x7f916c1600a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 --2- 172.21.15.1:0/2511275481 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f916c1615f0 0x7f916c1f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 >> 172.21.15.1:0/2511275481 conn(0x7f916c0d3ef0 msgr2=0x7f916c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 shutdown_connections 2024-03-24T10:33:57.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.140+0000 7f917202a640 1 -- 172.21.15.1:0/2511275481 wait complete. 2024-03-24T10:33:57.154 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T10:33:57.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T10:33:57.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.304+0000 7fb9a2f8a640 1 Processor -- start 2024-03-24T10:33:57.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.304+0000 7fb9a2f8a640 1 -- start start 2024-03-24T10:33:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a2f8a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb99c0cd270 0x7fb99c0cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a2f8a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb99c0b1740 con 0x7fb99c15c060 2024-03-24T10:33:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a2f8a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb99c0d2a90 con 0x7fb99c0d05f0 2024-03-24T10:33:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a2f8a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb99c15c440 con 0x7fb99c0cd270 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a0d26640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb99c15c060 0x7fb99c0cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39566/0 (socket says 172.21.15.1:39566) 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a0d26640 1 -- 172.21.15.1:0/1991895833 learned_addr learned my addr 172.21.15.1:0/1991895833 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.305+0000 7fb9a1527640 1 --2- 172.21.15.1:0/1991895833 >> v2:172.21.15.3:3300/0 conn(0x7fb99c0cd270 0x7fb99c0cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630140653 0 0) 0x7fb99c0b1740 con 0x7fb99c15c060 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb988003520 con 0x7fb99c15c060 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228449189 0 0) 0x7fb99c0d2a90 con 0x7fb99c0d05f0 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb99c0b1740 con 0x7fb99c0d05f0 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb9a1527640 1 -- 172.21.15.1:0/1991895833 >> v1:172.21.15.3:6789/0 conn(0x7fb99c0d05f0 legacy=0x7fb99c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb9a1527640 1 -- 172.21.15.1:0/1991895833 >> v1:172.21.15.1:6789/0 conn(0x7fb99c15c060 legacy=0x7fb99c0cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.306+0000 7fb9a1527640 1 -- 172.21.15.1:0/1991895833 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c0cd7c0 con 0x7fb99c0cd270 2024-03-24T10:33:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a1527640 1 --2- 172.21.15.1:0/1991895833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0cf820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9840097a0 tx=0x7fb98402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a718734bd13c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb984030610 con 0x7fb99c0cd270 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb984030770 con 0x7fb99c0cd270 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb99b7fe640 1 -- 172.21.15.1:0/1991895833 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb984030b00 con 0x7fb99c0cd270 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/1991895833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 msgr2=0x7fb99c0cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/1991895833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0cf820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9840097a0 tx=0x7fb98402b960 comp rx=0 tx=0).stop 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/1991895833 shutdown_connections 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/1991895833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0cf820 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/1991895833 >> 172.21.15.1:0/1991895833 conn(0x7fb99c0d1c50 msgr2=0x7fb99c0d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/1991895833 shutdown_connections 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.307+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/1991895833 wait complete. 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.308+0000 7fb9a2f8a640 1 Processor -- start 2024-03-24T10:33:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.308+0000 7fb9a2f8a640 1 -- start start 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb99c15c060 0x7fb99c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c0c8d20 con 0x7fb99c0d05f0 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c0c98a0 con 0x7fb99c0cd270 2024-03-24T10:33:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb9a2f8a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb99c0c7a50 con 0x7fb99c15c060 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45318/0 (socket says 172.21.15.1:45318) 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.309+0000 7fb99bfff640 1 -- 172.21.15.1:0/250946094 learned_addr learned my addr 172.21.15.1:0/250946094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9a0d26640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9a1527640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb99c15c060 0x7fb99c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 -- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb99c15c060 msgr2=0x7fb99c0c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb99c15c060 0x7fb99c0c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 -- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 msgr2=0x7fb99c0c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 -- 172.21.15.1:0/250946094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb984009400 con 0x7fb99c0d05f0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb99bfff640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00c8a0 tx=0x7fb98c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c003b70 con 0x7fb99c0d05f0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb98c003cf0 con 0x7fb99c0d05f0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c00aa90 con 0x7fb99c0d05f0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c0c8f70 con 0x7fb99c0d05f0 2024-03-24T10:33:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.310+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c0c9530 con 0x7fb99c0d05f0 2024-03-24T10:33:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.311+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c0b0e30 con 0x7fb99c0d05f0 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.312+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb98c00abf0 con 0x7fb99c0d05f0 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.312+0000 7fb9997fa640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 0x7fb97c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.312+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb98c089f00 con 0x7fb99c0d05f0 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.312+0000 7fb9a0d26640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 0x7fb97c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.313+0000 7fb9a0d26640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 0x7fb97c06dcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9940011e0 tx=0x7fb994007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.316+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb98c054a60 con 0x7fb99c0d05f0 2024-03-24T10:33:57.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:57.540+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb99c04f470 con 0x7fb99c0d05f0 2024-03-24T10:33:58.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.138+0000 7fb9997fa640 1 -- 172.21.15.1:0/250946094 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fb98c003e60 con 0x7fb99c0d05f0 2024-03-24T10:33:58.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 msgr2=0x7fb97c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 0x7fb97c06dcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9940011e0 tx=0x7fb994007d30 comp rx=0 tx=0).stop 2024-03-24T10:33:58.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 msgr2=0x7fb99c0c5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00c8a0 tx=0x7fb98c00cd60 comp rx=0 tx=0).stop 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 shutdown_connections 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb97c06b810 0x7fb97c06dcd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb99c15c060 0x7fb99c0c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb99c0d05f0 0x7fb99c0c5780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 --2- 172.21.15.1:0/250946094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb99c0cd270 0x7fb99c0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 >> 172.21.15.1:0/250946094 conn(0x7fb99c0d1c50 msgr2=0x7fb99c0cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 shutdown_connections 2024-03-24T10:33:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.141+0000 7fb9a2f8a640 1 -- 172.21.15.1:0/250946094 wait complete. 2024-03-24T10:33:58.155 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T10:33:58.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:33:58.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.259+0000 7f7be941c640 1 Processor -- start 2024-03-24T10:33:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.259+0000 7f7be941c640 1 -- start start 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be941c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7be4144b90 0x7f7be40cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be941c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7be4145260 0x7f7be406e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be941c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be4144f70 con 0x7f7be406e8a0 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be941c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7be406fbc0 con 0x7f7be4144b90 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be941c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f7be406de80 con 0x7f7be4145260 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7be4144b90 0x7f7be40cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7be406e8a0 0x7f7be4071c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39576/0 (socket says 172.21.15.1:39576) 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7be4144b90 0x7f7be40cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45330/0 (socket says 172.21.15.1:45330) 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be37fe640 1 -- 172.21.15.1:0/2252864196 learned_addr learned my addr 172.21.15.1:0/2252864196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.260+0000 7f7be27fc640 1 --2- 172.21.15.1:0/2252864196 >> v2:172.21.15.3:3300/0 conn(0x7f7be4145260 0x7f7be406e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be2ffd640 1 -- 172.21.15.1:0/2252864196 >> v2:172.21.15.3:3300/0 conn(0x7f7be4145260 msgr2=0x7f7be406e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be2ffd640 1 --2- 172.21.15.1:0/2252864196 >> v2:172.21.15.3:3300/0 conn(0x7f7be4145260 0x7f7be406e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be2ffd640 1 -- 172.21.15.1:0/2252864196 >> v1:172.21.15.1:6789/0 conn(0x7f7be406e8a0 legacy=0x7f7be4071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be2ffd640 1 -- 172.21.15.1:0/2252864196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be4160770 con 0x7f7be4144b90 2024-03-24T10:33:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be1ffb640 1 -- 172.21.15.1:0/2252864196 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837663183 0 0) 0x7f7be4144f70 con 0x7f7be406e8a0 2024-03-24T10:33:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be2ffd640 1 --2- 172.21.15.1:0/2252864196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be40cf300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7be4071810 tx=0x7f7bd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90df5878aad79c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be1ffb640 1 -- 172.21.15.1:0/2252864196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd402cc80 con 0x7f7be4144b90 2024-03-24T10:33:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be1ffb640 1 -- 172.21.15.1:0/2252864196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd402cde0 con 0x7f7be4144b90 2024-03-24T10:33:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.261+0000 7f7be1ffb640 1 -- 172.21.15.1:0/2252864196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4034660 con 0x7f7be4144b90 2024-03-24T10:33:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 -- 172.21.15.1:0/2252864196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 msgr2=0x7f7be40cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 --2- 172.21.15.1:0/2252864196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be40cf300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7be4071810 tx=0x7f7bd402bed0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 -- 172.21.15.1:0/2252864196 shutdown_connections 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 --2- 172.21.15.1:0/2252864196 >> v2:172.21.15.3:3300/0 conn(0x7f7be4145260 0x7f7be406e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 --2- 172.21.15.1:0/2252864196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be40cf300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 -- 172.21.15.1:0/2252864196 >> 172.21.15.1:0/2252864196 conn(0x7f7be40b2890 msgr2=0x7f7be40b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 -- 172.21.15.1:0/2252864196 shutdown_connections 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.262+0000 7f7be941c640 1 -- 172.21.15.1:0/2252864196 wait complete. 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.263+0000 7f7be941c640 1 Processor -- start 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.263+0000 7f7be941c640 1 -- start start 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7be406e8a0 0x7f7be414e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7be4145260 0x7f7be413a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7be4158bd0 con 0x7f7be4144b90 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7be413dfe0 con 0x7f7be406e8a0 2024-03-24T10:33:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be941c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7be413e2c0 con 0x7f7be4145260 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45338/0 (socket says 172.21.15.1:45338) 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be27fc640 1 -- 172.21.15.1:0/2366011534 learned_addr learned my addr 172.21.15.1:0/2366011534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be37fe640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7be4145260 0x7f7be413a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.264+0000 7f7be2ffd640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7be406e8a0 0x7f7be414e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 -- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7be4145260 msgr2=0x7f7be413a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7be4145260 0x7f7be413a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 -- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7be406e8a0 msgr2=0x7f7be414e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7be406e8a0 0x7f7be414e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 -- 172.21.15.1:0/2366011534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd4009a70 con 0x7f7be4144b90 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be27fc640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00e870 tx=0x7f7bcc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0094f0 con 0x7f7be4144b90 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bcc00c940 con 0x7f7be4144b90 2024-03-24T10:33:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be413e5a0 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00ccf0 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.265+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be413eb60 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.266+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be406f080 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.266+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7bcc013470 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.267+0000 7f7bdb7fe640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 0x7f7bc406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.267+0000 7f7be2ffd640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 0x7f7bc406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.267+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7bcc08a1f0 con 0x7f7be4144b90 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.267+0000 7f7be2ffd640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 0x7f7bc406dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.271+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7bcc054d50 con 0x7f7be4144b90 2024-03-24T10:33:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.488+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7be404f4e0 con 0x7f7be4144b90 2024-03-24T10:33:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.488+0000 7f7bdb7fe640 1 -- 172.21.15.1:0/2366011534 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f7bcc058410 con 0x7f7be4144b90 2024-03-24T10:33:58.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:58.490 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:33:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 msgr2=0x7f7bc406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 0x7f7bc406dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036000 tx=0x7f7bd4030000 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 msgr2=0x7f7be413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00e870 tx=0x7f7bcc00ed30 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 shutdown_connections 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7bc406b810 0x7f7bc406dcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7be4145260 0x7f7be413a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7be4144b90 0x7f7be413f080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 --2- 172.21.15.1:0/2366011534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7be406e8a0 0x7f7be414e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 >> 172.21.15.1:0/2366011534 conn(0x7f7be40b2890 msgr2=0x7f7be4141800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 shutdown_connections 2024-03-24T10:33:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.490+0000 7f7be941c640 1 -- 172.21.15.1:0/2366011534 wait complete. 2024-03-24T10:33:58.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:58.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.603+0000 7fac2aba2640 1 Processor -- start 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.603+0000 7fac2aba2640 1 -- start start 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2aba2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac240d0c10 0x7fac240b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2aba2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac240b4430 0x7fac240b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2aba2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac240cfbe0 con 0x7fac240b4050 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2aba2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fac240d48e0 con 0x7fac240b4430 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2aba2640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fac240d4bc0 con 0x7fac240d0c10 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2a3a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac240b4430 0x7fac240b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac29ba0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac240b4050 0x7fac240d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39590/0 (socket says 172.21.15.1:39590) 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac29ba0640 1 -- 172.21.15.1:0/3435861398 learned_addr learned my addr 172.21.15.1:0/3435861398 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2939f640 1 --2- 172.21.15.1:0/3435861398 >> v2:172.21.15.3:3300/0 conn(0x7fac240d0c10 0x7fac240b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2a3a1640 1 -- 172.21.15.1:0/3435861398 >> v2:172.21.15.3:3300/0 conn(0x7fac240d0c10 msgr2=0x7fac240b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2a3a1640 1 --2- 172.21.15.1:0/3435861398 >> v2:172.21.15.3:3300/0 conn(0x7fac240d0c10 0x7fac240b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2a3a1640 1 -- 172.21.15.1:0/3435861398 >> v1:172.21.15.1:6789/0 conn(0x7fac240b4050 legacy=0x7fac240d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac28b9e640 1 -- 172.21.15.1:0/3435861398 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887450844 0 0) 0x7fac240cfbe0 con 0x7fac240b4050 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.604+0000 7fac2a3a1640 1 -- 172.21.15.1:0/3435861398 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac240cf590 con 0x7fac240b4430 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.605+0000 7fac2a3a1640 1 --2- 172.21.15.1:0/3435861398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240b4430 0x7fac240b71b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac2408e510 tx=0x7fac1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7575da1546b5c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.605+0000 7fac28b9e640 1 -- 172.21.15.1:0/3435861398 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14031830 con 0x7fac240b4430 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.605+0000 7fac28b9e640 1 -- 172.21.15.1:0/3435861398 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac14031990 con 0x7fac240b4430 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.605+0000 7fac28b9e640 1 -- 172.21.15.1:0/3435861398 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14031d20 con 0x7fac240b4430 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 -- 172.21.15.1:0/3435861398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240b4430 msgr2=0x7fac240b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 --2- 172.21.15.1:0/3435861398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240b4430 0x7fac240b71b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac2408e510 tx=0x7fac1402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 -- 172.21.15.1:0/3435861398 shutdown_connections 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 --2- 172.21.15.1:0/3435861398 >> v2:172.21.15.3:3300/0 conn(0x7fac240d0c10 0x7fac240b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 --2- 172.21.15.1:0/3435861398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240b4430 0x7fac240b71b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 -- 172.21.15.1:0/3435861398 >> 172.21.15.1:0/3435861398 conn(0x7fac240d3d20 msgr2=0x7fac240d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 -- 172.21.15.1:0/3435861398 shutdown_connections 2024-03-24T10:33:58.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.606+0000 7fac2aba2640 1 -- 172.21.15.1:0/3435861398 wait complete. 2024-03-24T10:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.607+0000 7fac2aba2640 1 Processor -- start 2024-03-24T10:33:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.607+0000 7fac2aba2640 1 -- start start 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac240b4050 0x7fac240b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac240b4430 0x7fac240d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fac241f35f0 con 0x7fac240d0c10 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fac240c8420 con 0x7fac240b4050 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2aba2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fac240c86e0 con 0x7fac240b4430 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45358/0 (socket says 172.21.15.1:45358) 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 -- 172.21.15.1:0/1354731483 learned_addr learned my addr 172.21.15.1:0/1354731483 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac29ba0640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac240b4050 0x7fac240b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2939f640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac240b4430 0x7fac240d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 -- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac240b4430 msgr2=0x7fac240d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac240b4430 0x7fac240d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 -- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac240b4050 msgr2=0x7fac240b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac240b4050 0x7fac240b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 -- 172.21.15.1:0/1354731483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac14009400 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac2a3a1640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fac1402bee0 tx=0x7fac140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1403a630 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.608+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1403a7b0 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.609+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac240c89c0 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.609+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1403ab60 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.609+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac240c8f80 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.610+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac240d2c90 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.610+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac1403acc0 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.610+0000 7fac227fc640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 0x7fac1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.610+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac140abc80 con 0x7fac240d0c10 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.610+0000 7fac29ba0640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 0x7fac1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.611+0000 7fac29ba0640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 0x7fac1006dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0012e0 tx=0x7fac1c001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.614+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac14076750 con 0x7fac240d0c10 2024-03-24T10:33:58.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.800+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac2404f470 con 0x7fac240d0c10 2024-03-24T10:33:58.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.802+0000 7fac227fc640 1 -- 172.21.15.1:0/1354731483 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fac14035020 con 0x7fac240d0c10 2024-03-24T10:33:58.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:58.803 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:33:58.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 msgr2=0x7fac1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 0x7fac1006dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0012e0 tx=0x7fac1c001630 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 msgr2=0x7fac240d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fac1402bee0 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 shutdown_connections 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac1006b810 0x7fac1006dcd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac240d0c10 0x7fac240d23d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac240b4430 0x7fac240d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 --2- 172.21.15.1:0/1354731483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac240b4050 0x7fac240b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 >> 172.21.15.1:0/1354731483 conn(0x7fac240d3d20 msgr2=0x7fac240b74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 shutdown_connections 2024-03-24T10:33:58.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.804+0000 7fac2aba2640 1 -- 172.21.15.1:0/1354731483 wait complete. 2024-03-24T10:33:58.816 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T10:33:58.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:33:58.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.916+0000 7fb937df4640 1 Processor -- start 2024-03-24T10:33:58.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.916+0000 7fb937df4640 1 -- start start 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 0x7fb9300d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb93006c770 0x7fb93006cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb93006d100 0x7fb93015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb93006d5b0 con 0x7fb93006d100 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb9300d1340 con 0x7fb9300b74c0 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb937df4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb9300b4c70 con 0x7fb93006c770 2024-03-24T10:33:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb93006d100 0x7fb93015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb93006d100 0x7fb93015d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45362/0 (socket says 172.21.15.1:45362) 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 -- 172.21.15.1:0/917263405 learned_addr learned my addr 172.21.15.1:0/917263405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb935b90640 1 --2- 172.21.15.1:0/917263405 >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 0x7fb9300d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb93538f640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb93006c770 0x7fb93006cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 -- 172.21.15.1:0/917263405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb93006c770 msgr2=0x7fb93006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb93006c770 0x7fb93006cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 -- 172.21.15.1:0/917263405 >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 msgr2=0x7fb9300d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 --2- 172.21.15.1:0/917263405 >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 0x7fb9300d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb936391640 1 -- 172.21.15.1:0/917263405 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9300b4f50 con 0x7fb93006d100 2024-03-24T10:33:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.917+0000 7fb935b90640 1 --2- 172.21.15.1:0/917263405 >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 0x7fb9300d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.918+0000 7fb936391640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015d8f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb93006bba0 tx=0x7fb92802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7968873395693ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.918+0000 7fb934b8e640 1 -- 172.21.15.1:0/917263405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928031870 con 0x7fb93006d100 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.918+0000 7fb934b8e640 1 -- 172.21.15.1:0/917263405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9280319d0 con 0x7fb93006d100 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.918+0000 7fb934b8e640 1 -- 172.21.15.1:0/917263405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928031d60 con 0x7fb93006d100 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 -- 172.21.15.1:0/917263405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 msgr2=0x7fb93015d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015d8f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb93006bba0 tx=0x7fb92802b910 comp rx=0 tx=0).stop 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 -- 172.21.15.1:0/917263405 shutdown_connections 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015d8f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 --2- 172.21.15.1:0/917263405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb93006c770 0x7fb93006cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 --2- 172.21.15.1:0/917263405 >> v2:172.21.15.3:3300/0 conn(0x7fb9300b74c0 0x7fb9300d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 -- 172.21.15.1:0/917263405 >> 172.21.15.1:0/917263405 conn(0x7fb9300d1c80 msgr2=0x7fb9300d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 -- 172.21.15.1:0/917263405 shutdown_connections 2024-03-24T10:33:58.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.919+0000 7fb937df4640 1 -- 172.21.15.1:0/917263405 wait complete. 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.920+0000 7fb937df4640 1 Processor -- start 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.920+0000 7fb937df4640 1 -- start start 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb93006c770 0x7fb93015d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9300b74c0 0x7fb9301582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb93015db10 con 0x7fb93006d100 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb93015c7b0 con 0x7fb93006c770 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb9300c6050 con 0x7fb9300b74c0 2024-03-24T10:33:58.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45376/0 (socket says 172.21.15.1:45376) 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb935b90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb93006c770 0x7fb93015d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 -- 172.21.15.1:0/3711630194 learned_addr learned my addr 172.21.15.1:0/3711630194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb936391640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9300b74c0 0x7fb9301582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 -- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9300b74c0 msgr2=0x7fb9301582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9300b74c0 0x7fb9301582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 -- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb93006c770 msgr2=0x7fb93015d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb93006c770 0x7fb93015d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 -- 172.21.15.1:0/3711630194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c009a70 con 0x7fb93006d100 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb93538f640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb92000e870 tx=0x7fb92000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9200094f0 con 0x7fb93006d100 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb92000c940 con 0x7fb93006d100 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92000ccf0 con 0x7fb93006d100 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9280094d0 con 0x7fb93006d100 2024-03-24T10:33:58.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.921+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9300c6610 con 0x7fb93006d100 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.922+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb93004f4e0 con 0x7fb93006d100 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.923+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb920013470 con 0x7fb93006d100 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.923+0000 7fb9267fc640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 0x7fb91406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.923+0000 7fb935b90640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 0x7fb91406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.923+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb92008a2b0 con 0x7fb93006d100 2024-03-24T10:33:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.924+0000 7fb935b90640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 0x7fb91406dd00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009fd0 tx=0x7fb92c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:58.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:58.928+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb920054d00 con 0x7fb93006d100 2024-03-24T10:33:59.097 INFO:tasks.ceph.mds.b.smithi003.stderr:2024-03-24T10:33:59.096+0000 7f5f1fcd8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:33:59.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.104+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9300b5510 con 0x7fb91406b840 2024-03-24T10:33:59.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.111+0000 7fb9267fc640 1 -- 172.21.15.1:0/3711630194 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7fb92c00f040 con 0x7fb91406b840 2024-03-24T10:33:59.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:59.116 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:33:59.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 msgr2=0x7fb91406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 0x7fb91406dd00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009fd0 tx=0x7fb92c00b040 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 msgr2=0x7fb93015c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb92000e870 tx=0x7fb92000ed30 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 shutdown_connections 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91406b840 0x7fb91406dd00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb9300b74c0 0x7fb9301582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93006d100 0x7fb93015c270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 --2- 172.21.15.1:0/3711630194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb93006c770 0x7fb93015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 >> 172.21.15.1:0/3711630194 conn(0x7fb9300d1c80 msgr2=0x7fb9300b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 shutdown_connections 2024-03-24T10:33:59.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.114+0000 7fb937df4640 1 -- 172.21.15.1:0/3711630194 wait complete. 2024-03-24T10:33:59.126 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-24T10:33:58.517982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235656,"kb_used_data":2848,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739064,"statfs":{"total":773094113280,"available":772852801536,"internally_reserved":0,"allocated":2916352,"data_stored":1752596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412770"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306747+0000","last_change":"2024-03-24T10:33:55.160859+0000","last_active":"2024-03-24T10:33:56.306747+0000","last_peered":"2024-03-24T10:33:56.306747+0000","last_clean":"2024-03-24T10:33:56.306747+0000","last_became_active":"2024-03-24T10:33:55.160069+0000","last_became_peered":"2024-03-24T10:33:55.160069+0000","last_unstale":"2024-03-24T10:33:56.306747+0000","last_undegraded":"2024-03-24T10:33:56.306747+0000","last_fullsized":"2024-03-24T10:33:56.306747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123102+0000","last_change":"2024-03-24T10:33:55.140972+0000","last_active":"2024-03-24T10:33:56.123102+0000","last_peered":"2024-03-24T10:33:56.123102+0000","last_clean":"2024-03-24T10:33:56.123102+0000","last_became_active":"2024-03-24T10:33:55.140573+0000","last_became_peered":"2024-03-24T10:33:55.140573+0000","last_unstale":"2024-03-24T10:33:56.123102+0000","last_undegraded":"2024-03-24T10:33:56.123102+0000","last_fullsized":"2024-03-24T10:33:56.123102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305754+0000","last_change":"2024-03-24T10:33:55.141078+0000","last_active":"2024-03-24T10:33:56.305754+0000","last_peered":"2024-03-24T10:33:56.305754+0000","last_clean":"2024-03-24T10:33:56.305754+0000","last_became_active":"2024-03-24T10:33:55.140774+0000","last_became_peered":"2024-03-24T10:33:55.140774+0000","last_unstale":"2024-03-24T10:33:56.305754+0000","last_undegraded":"2024-03-24T10:33:56.305754+0000","last_fullsized":"2024-03-24T10:33:56.305754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305846+0000","last_change":"2024-03-24T10:33:55.148019+0000","last_active":"2024-03-24T10:33:56.305846+0000","last_peered":"2024-03-24T10:33:56.305846+0000","last_clean":"2024-03-24T10:33:56.305846+0000","last_became_active":"2024-03-24T10:33:55.145313+0000","last_became_peered":"2024-03-24T10:33:55.145313+0000","last_unstale":"2024-03-24T10:33:56.305846+0000","last_undegraded":"2024-03-24T10:33:56.305846+0000","last_fullsized":"2024-03-24T10:33:56.305846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306869+0000","last_change":"2024-03-24T10:33:55.145334+0000","last_active":"2024-03-24T10:33:56.306869+0000","last_peered":"2024-03-24T10:33:56.306869+0000","last_clean":"2024-03-24T10:33:56.306869+0000","last_became_active":"2024-03-24T10:33:55.145018+0000","last_became_peered":"2024-03-24T10:33:55.145018+0000","last_unstale":"2024-03-24T10:33:56.306869+0000","last_undegraded":"2024-03-24T10:33:56.306869+0000","last_fullsized":"2024-03-24T10:33:56.306869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:24:24.292025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305901+0000","last_change":"2024-03-24T10:33:55.142392+0000","last_active":"2024-03-24T10:33:56.305901+0000","last_peered":"2024-03-24T10:33:56.305901+0000","last_clean":"2024-03-24T10:33:56.305901+0000","last_became_active":"2024-03-24T10:33:55.142070+0000","last_became_peered":"2024-03-24T10:33:55.142070+0000","last_unstale":"2024-03-24T10:33:56.305901+0000","last_undegraded":"2024-03-24T10:33:56.305901+0000","last_fullsized":"2024-03-24T10:33:56.305901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306206+0000","last_change":"2024-03-24T10:33:55.158958+0000","last_active":"2024-03-24T10:33:56.306206+0000","last_peered":"2024-03-24T10:33:56.306206+0000","last_clean":"2024-03-24T10:33:56.306206+0000","last_became_active":"2024-03-24T10:33:55.158270+0000","last_became_peered":"2024-03-24T10:33:55.158270+0000","last_unstale":"2024-03-24T10:33:56.306206+0000","last_undegraded":"2024-03-24T10:33:56.306206+0000","last_fullsized":"2024-03-24T10:33:56.306206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305233+0000","last_change":"2024-03-24T10:33:55.142337+0000","last_active":"2024-03-24T10:33:56.305233+0000","last_peered":"2024-03-24T10:33:56.305233+0000","last_clean":"2024-03-24T10:33:56.305233+0000","last_became_active":"2024-03-24T10:33:55.141994+0000","last_became_peered":"2024-03-24T10:33:55.141994+0000","last_unstale":"2024-03-24T10:33:56.305233+0000","last_undegraded":"2024-03-24T10:33:56.305233+0000","last_fullsized":"2024-03-24T10:33:56.305233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306802+0000","last_change":"2024-03-24T10:33:55.155886+0000","last_active":"2024-03-24T10:33:56.306802+0000","last_peered":"2024-03-24T10:33:56.306802+0000","last_clean":"2024-03-24T10:33:56.306802+0000","last_became_active":"2024-03-24T10:33:55.153792+0000","last_became_peered":"2024-03-24T10:33:55.153792+0000","last_unstale":"2024-03-24T10:33:56.306802+0000","last_undegraded":"2024-03-24T10:33:56.306802+0000","last_fullsized":"2024-03-24T10:33:56.306802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124307+0000","last_change":"2024-03-24T10:33:55.152791+0000","last_active":"2024-03-24T10:33:56.124307+0000","last_peered":"2024-03-24T10:33:56.124307+0000","last_clean":"2024-03-24T10:33:56.124307+0000","last_became_active":"2024-03-24T10:33:55.152524+0000","last_became_peered":"2024-03-24T10:33:55.152524+0000","last_unstale":"2024-03-24T10:33:56.124307+0000","last_undegraded":"2024-03-24T10:33:56.124307+0000","last_fullsized":"2024-03-24T10:33:56.124307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124236+0000","last_change":"2024-03-24T10:33:55.153894+0000","last_active":"2024-03-24T10:33:56.124236+0000","last_peered":"2024-03-24T10:33:56.124236+0000","last_clean":"2024-03-24T10:33:56.124236+0000","last_became_active":"2024-03-24T10:33:55.153623+0000","last_became_peered":"2024-03-24T10:33:55.153623+0000","last_unstale":"2024-03-24T10:33:56.124236+0000","last_undegraded":"2024-03-24T10:33:56.124236+0000","last_fullsized":"2024-03-24T10:33:56.124236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306325+0000","last_change":"2024-03-24T10:33:55.148337+0000","last_active":"2024-03-24T10:33:56.306325+0000","last_peered":"2024-03-24T10:33:56.306325+0000","last_clean":"2024-03-24T10:33:56.306325+0000","last_became_active":"2024-03-24T10:33:55.148076+0000","last_became_peered":"2024-03-24T10:33:55.148076+0000","last_unstale":"2024-03-24T10:33:56.306325+0000","last_undegraded":"2024-03-24T10:33:56.306325+0000","last_fullsized":"2024-03-24T10:33:56.306325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305500+0000","last_change":"2024-03-24T10:33:55.148625+0000","last_active":"2024-03-24T10:33:56.305500+0000","last_peered":"2024-03-24T10:33:56.305500+0000","last_clean":"2024-03-24T10:33:56.305500+0000","last_became_active":"2024-03-24T10:33:55.148234+0000","last_became_peered":"2024-03-24T10:33:55.148234+0000","last_unstale":"2024-03-24T10:33:56.305500+0000","last_undegraded":"2024-03-24T10:33:56.305500+0000","last_fullsized":"2024-03-24T10:33:56.305500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305594+0000","last_change":"2024-03-24T10:33:55.141825+0000","last_active":"2024-03-24T10:33:56.305594+0000","last_peered":"2024-03-24T10:33:56.305594+0000","last_clean":"2024-03-24T10:33:56.305594+0000","last_became_active":"2024-03-24T10:33:55.141380+0000","last_became_peered":"2024-03-24T10:33:55.141380+0000","last_unstale":"2024-03-24T10:33:56.305594+0000","last_undegraded":"2024-03-24T10:33:56.305594+0000","last_fullsized":"2024-03-24T10:33:56.305594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:51.274453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122775+0000","last_change":"2024-03-24T10:33:55.132926+0000","last_active":"2024-03-24T10:33:56.122775+0000","last_peered":"2024-03-24T10:33:56.122775+0000","last_clean":"2024-03-24T10:33:56.122775+0000","last_became_active":"2024-03-24T10:33:55.132614+0000","last_became_peered":"2024-03-24T10:33:55.132614+0000","last_unstale":"2024-03-24T10:33:56.122775+0000","last_undegraded":"2024-03-24T10:33:56.122775+0000","last_fullsized":"2024-03-24T10:33:56.122775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305340+0000","last_change":"2024-03-24T10:33:55.141440+0000","last_active":"2024-03-24T10:33:56.305340+0000","last_peered":"2024-03-24T10:33:56.305340+0000","last_clean":"2024-03-24T10:33:56.305340+0000","last_became_active":"2024-03-24T10:33:55.141096+0000","last_became_peered":"2024-03-24T10:33:55.141096+0000","last_unstale":"2024-03-24T10:33:56.305340+0000","last_undegraded":"2024-03-24T10:33:56.305340+0000","last_fullsized":"2024-03-24T10:33:56.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306420+0000","last_change":"2024-03-24T10:33:55.152651+0000","last_active":"2024-03-24T10:33:56.306420+0000","last_peered":"2024-03-24T10:33:56.306420+0000","last_clean":"2024-03-24T10:33:56.306420+0000","last_became_active":"2024-03-24T10:33:55.152386+0000","last_became_peered":"2024-03-24T10:33:55.152386+0000","last_unstale":"2024-03-24T10:33:56.306420+0000","last_undegraded":"2024-03-24T10:33:56.306420+0000","last_fullsized":"2024-03-24T10:33:56.306420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123835+0000","last_change":"2024-03-24T10:33:55.149184+0000","last_active":"2024-03-24T10:33:56.123835+0000","last_peered":"2024-03-24T10:33:56.123835+0000","last_clean":"2024-03-24T10:33:56.123835+0000","last_became_active":"2024-03-24T10:33:55.148828+0000","last_became_peered":"2024-03-24T10:33:55.148828+0000","last_unstale":"2024-03-24T10:33:56.123835+0000","last_undegraded":"2024-03-24T10:33:56.123835+0000","last_fullsized":"2024-03-24T10:33:56.123835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:33.564193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306947+0000","last_change":"2024-03-24T10:33:55.147090+0000","last_active":"2024-03-24T10:33:56.306947+0000","last_peered":"2024-03-24T10:33:56.306947+0000","last_clean":"2024-03-24T10:33:56.306947+0000","last_became_active":"2024-03-24T10:33:55.146820+0000","last_became_peered":"2024-03-24T10:33:55.146820+0000","last_unstale":"2024-03-24T10:33:56.306947+0000","last_undegraded":"2024-03-24T10:33:56.306947+0000","last_fullsized":"2024-03-24T10:33:56.306947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:12.251747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124127+0000","last_change":"2024-03-24T10:33:55.156434+0000","last_active":"2024-03-24T10:33:56.124127+0000","last_peered":"2024-03-24T10:33:56.124127+0000","last_clean":"2024-03-24T10:33:56.124127+0000","last_became_active":"2024-03-24T10:33:55.156176+0000","last_became_peered":"2024-03-24T10:33:55.156176+0000","last_unstale":"2024-03-24T10:33:56.124127+0000","last_undegraded":"2024-03-24T10:33:56.124127+0000","last_fullsized":"2024-03-24T10:33:56.124127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305688+0000","last_change":"2024-03-24T10:33:55.155986+0000","last_active":"2024-03-24T10:33:56.305688+0000","last_peered":"2024-03-24T10:33:56.305688+0000","last_clean":"2024-03-24T10:33:56.305688+0000","last_became_active":"2024-03-24T10:33:55.155699+0000","last_became_peered":"2024-03-24T10:33:55.155699+0000","last_unstale":"2024-03-24T10:33:56.305688+0000","last_undegraded":"2024-03-24T10:33:56.305688+0000","last_fullsized":"2024-03-24T10:33:56.305688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:41.606563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305535+0000","last_change":"2024-03-24T10:33:55.151865+0000","last_active":"2024-03-24T10:33:56.305535+0000","last_peered":"2024-03-24T10:33:56.305535+0000","last_clean":"2024-03-24T10:33:56.305535+0000","last_became_active":"2024-03-24T10:33:55.151592+0000","last_became_peered":"2024-03-24T10:33:55.151592+0000","last_unstale":"2024-03-24T10:33:56.305535+0000","last_undegraded":"2024-03-24T10:33:56.305535+0000","last_fullsized":"2024-03-24T10:33:56.305535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305309+0000","last_change":"2024-03-24T10:33:55.152238+0000","last_active":"2024-03-24T10:33:56.305309+0000","last_peered":"2024-03-24T10:33:56.305309+0000","last_clean":"2024-03-24T10:33:56.305309+0000","last_became_active":"2024-03-24T10:33:55.151966+0000","last_became_peered":"2024-03-24T10:33:55.151966+0000","last_unstale":"2024-03-24T10:33:56.305309+0000","last_undegraded":"2024-03-24T10:33:56.305309+0000","last_fullsized":"2024-03-24T10:33:56.305309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305757+0000","last_change":"2024-03-24T10:33:55.144173+0000","last_active":"2024-03-24T10:33:56.305757+0000","last_peered":"2024-03-24T10:33:56.305757+0000","last_clean":"2024-03-24T10:33:56.305757+0000","last_became_active":"2024-03-24T10:33:55.143874+0000","last_became_peered":"2024-03-24T10:33:55.143874+0000","last_unstale":"2024-03-24T10:33:56.305757+0000","last_undegraded":"2024-03-24T10:33:56.305757+0000","last_fullsized":"2024-03-24T10:33:56.305757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.136315+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.135994+0000","last_became_peered":"2024-03-24T10:33:55.135994+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124458+0000","last_change":"2024-03-24T10:33:55.159140+0000","last_active":"2024-03-24T10:33:56.124458+0000","last_peered":"2024-03-24T10:33:56.124458+0000","last_clean":"2024-03-24T10:33:56.124458+0000","last_became_active":"2024-03-24T10:33:55.158875+0000","last_became_peered":"2024-03-24T10:33:55.158875+0000","last_unstale":"2024-03-24T10:33:56.124458+0000","last_undegraded":"2024-03-24T10:33:56.124458+0000","last_fullsized":"2024-03-24T10:33:56.124458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123157+0000","last_change":"2024-03-24T10:33:55.134527+0000","last_active":"2024-03-24T10:33:56.123157+0000","last_peered":"2024-03-24T10:33:56.123157+0000","last_clean":"2024-03-24T10:33:56.123157+0000","last_became_active":"2024-03-24T10:33:55.134220+0000","last_became_peered":"2024-03-24T10:33:55.134220+0000","last_unstale":"2024-03-24T10:33:56.123157+0000","last_undegraded":"2024-03-24T10:33:56.123157+0000","last_fullsized":"2024-03-24T10:33:56.123157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306047+0000","last_change":"2024-03-24T10:33:55.138237+0000","last_active":"2024-03-24T10:33:56.306047+0000","last_peered":"2024-03-24T10:33:56.306047+0000","last_clean":"2024-03-24T10:33:56.306047+0000","last_became_active":"2024-03-24T10:33:55.136755+0000","last_became_peered":"2024-03-24T10:33:55.136755+0000","last_unstale":"2024-03-24T10:33:56.306047+0000","last_undegraded":"2024-03-24T10:33:56.306047+0000","last_fullsized":"2024-03-24T10:33:56.306047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305840+0000","last_change":"2024-03-24T10:33:55.145372+0000","last_active":"2024-03-24T10:33:56.305840+0000","last_peered":"2024-03-24T10:33:56.305840+0000","last_clean":"2024-03-24T10:33:56.305840+0000","last_became_active":"2024-03-24T10:33:55.144207+0000","last_became_peered":"2024-03-24T10:33:55.144207+0000","last_unstale":"2024-03-24T10:33:56.305840+0000","last_undegraded":"2024-03-24T10:33:56.305840+0000","last_fullsized":"2024-03-24T10:33:56.305840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306010+0000","last_change":"2024-03-24T10:33:55.146001+0000","last_active":"2024-03-24T10:33:56.306010+0000","last_peered":"2024-03-24T10:33:56.306010+0000","last_clean":"2024-03-24T10:33:56.306010+0000","last_became_active":"2024-03-24T10:33:55.145472+0000","last_became_peered":"2024-03-24T10:33:55.145472+0000","last_unstale":"2024-03-24T10:33:56.306010+0000","last_undegraded":"2024-03-24T10:33:56.306010+0000","last_fullsized":"2024-03-24T10:33:56.306010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123635+0000","last_change":"2024-03-24T10:33:55.143320+0000","last_active":"2024-03-24T10:33:56.123635+0000","last_peered":"2024-03-24T10:33:56.123635+0000","last_clean":"2024-03-24T10:33:56.123635+0000","last_became_active":"2024-03-24T10:33:55.143046+0000","last_became_peered":"2024-03-24T10:33:55.143046+0000","last_unstale":"2024-03-24T10:33:56.123635+0000","last_undegraded":"2024-03-24T10:33:56.123635+0000","last_fullsized":"2024-03-24T10:33:56.123635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124493+0000","last_change":"2024-03-24T10:33:55.147275+0000","last_active":"2024-03-24T10:33:56.124493+0000","last_peered":"2024-03-24T10:33:56.124493+0000","last_clean":"2024-03-24T10:33:56.124493+0000","last_became_active":"2024-03-24T10:33:55.146476+0000","last_became_peered":"2024-03-24T10:33:55.146476+0000","last_unstale":"2024-03-24T10:33:56.124493+0000","last_undegraded":"2024-03-24T10:33:56.124493+0000","last_fullsized":"2024-03-24T10:33:56.124493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305716+0000","last_change":"2024-03-24T10:33:55.131787+0000","last_active":"2024-03-24T10:33:56.305716+0000","last_peered":"2024-03-24T10:33:56.305716+0000","last_clean":"2024-03-24T10:33:56.305716+0000","last_became_active":"2024-03-24T10:33:55.131462+0000","last_became_peered":"2024-03-24T10:33:55.131462+0000","last_unstale":"2024-03-24T10:33:56.305716+0000","last_undegraded":"2024-03-24T10:33:56.305716+0000","last_fullsized":"2024-03-24T10:33:56.305716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:24.900927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305842+0000","last_change":"2024-03-24T10:33:54.130171+0000","last_active":"2024-03-24T10:33:56.305842+0000","last_peered":"2024-03-24T10:33:56.305842+0000","last_clean":"2024-03-24T10:33:56.305842+0000","last_became_active":"2024-03-24T10:33:54.129845+0000","last_became_peered":"2024-03-24T10:33:54.129845+0000","last_unstale":"2024-03-24T10:33:56.305842+0000","last_undegraded":"2024-03-24T10:33:56.305842+0000","last_fullsized":"2024-03-24T10:33:56.305842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305988+0000","last_change":"2024-03-24T10:33:55.156576+0000","last_active":"2024-03-24T10:33:56.305988+0000","last_peered":"2024-03-24T10:33:56.305988+0000","last_clean":"2024-03-24T10:33:56.305988+0000","last_became_active":"2024-03-24T10:33:55.156301+0000","last_became_peered":"2024-03-24T10:33:55.156301+0000","last_unstale":"2024-03-24T10:33:56.305988+0000","last_undegraded":"2024-03-24T10:33:56.305988+0000","last_fullsized":"2024-03-24T10:33:56.305988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124113+0000","last_change":"2024-03-24T10:33:54.122838+0000","last_active":"2024-03-24T10:33:56.124113+0000","last_peered":"2024-03-24T10:33:56.124113+0000","last_clean":"2024-03-24T10:33:56.124113+0000","last_became_active":"2024-03-24T10:33:54.122012+0000","last_became_peered":"2024-03-24T10:33:54.122012+0000","last_unstale":"2024-03-24T10:33:56.124113+0000","last_undegraded":"2024-03-24T10:33:56.124113+0000","last_fullsized":"2024-03-24T10:33:56.124113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305341+0000","last_change":"2024-03-24T10:33:54.127097+0000","last_active":"2024-03-24T10:33:56.305341+0000","last_peered":"2024-03-24T10:33:56.305341+0000","last_clean":"2024-03-24T10:33:56.305341+0000","last_became_active":"2024-03-24T10:33:54.126575+0000","last_became_peered":"2024-03-24T10:33:54.126575+0000","last_unstale":"2024-03-24T10:33:56.305341+0000","last_undegraded":"2024-03-24T10:33:56.305341+0000","last_fullsized":"2024-03-24T10:33:56.305341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305318+0000","last_change":"2024-03-24T10:33:55.138621+0000","last_active":"2024-03-24T10:33:56.305318+0000","last_peered":"2024-03-24T10:33:56.305318+0000","last_clean":"2024-03-24T10:33:56.305318+0000","last_became_active":"2024-03-24T10:33:55.138352+0000","last_became_peered":"2024-03-24T10:33:55.138352+0000","last_unstale":"2024-03-24T10:33:56.305318+0000","last_undegraded":"2024-03-24T10:33:56.305318+0000","last_fullsized":"2024-03-24T10:33:56.305318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305437+0000","last_change":"2024-03-24T10:33:54.125263+0000","last_active":"2024-03-24T10:33:56.305437+0000","last_peered":"2024-03-24T10:33:56.305437+0000","last_clean":"2024-03-24T10:33:56.305437+0000","last_became_active":"2024-03-24T10:33:54.124945+0000","last_became_peered":"2024-03-24T10:33:54.124945+0000","last_unstale":"2024-03-24T10:33:56.305437+0000","last_undegraded":"2024-03-24T10:33:56.305437+0000","last_fullsized":"2024-03-24T10:33:56.305437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122873+0000","last_change":"2024-03-24T10:33:55.134395+0000","last_active":"2024-03-24T10:33:56.122873+0000","last_peered":"2024-03-24T10:33:56.122873+0000","last_clean":"2024-03-24T10:33:56.122873+0000","last_became_active":"2024-03-24T10:33:55.133957+0000","last_became_peered":"2024-03-24T10:33:55.133957+0000","last_unstale":"2024-03-24T10:33:56.122873+0000","last_undegraded":"2024-03-24T10:33:56.122873+0000","last_fullsized":"2024-03-24T10:33:56.122873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124381+0000","last_change":"2024-03-24T10:33:54.127218+0000","last_active":"2024-03-24T10:33:56.124381+0000","last_peered":"2024-03-24T10:33:56.124381+0000","last_clean":"2024-03-24T10:33:56.124381+0000","last_became_active":"2024-03-24T10:33:54.126932+0000","last_became_peered":"2024-03-24T10:33:54.126932+0000","last_unstale":"2024-03-24T10:33:56.124381+0000","last_undegraded":"2024-03-24T10:33:56.124381+0000","last_fullsized":"2024-03-24T10:33:56.124381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306066+0000","last_change":"2024-03-24T10:33:55.150999+0000","last_active":"2024-03-24T10:33:56.306066+0000","last_peered":"2024-03-24T10:33:56.306066+0000","last_clean":"2024-03-24T10:33:56.306066+0000","last_became_active":"2024-03-24T10:33:55.150729+0000","last_became_peered":"2024-03-24T10:33:55.150729+0000","last_unstale":"2024-03-24T10:33:56.306066+0000","last_undegraded":"2024-03-24T10:33:56.306066+0000","last_fullsized":"2024-03-24T10:33:56.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306163+0000","last_change":"2024-03-24T10:33:54.126346+0000","last_active":"2024-03-24T10:33:56.306163+0000","last_peered":"2024-03-24T10:33:56.306163+0000","last_clean":"2024-03-24T10:33:56.306163+0000","last_became_active":"2024-03-24T10:33:54.126026+0000","last_became_peered":"2024-03-24T10:33:54.126026+0000","last_unstale":"2024-03-24T10:33:56.306163+0000","last_undegraded":"2024-03-24T10:33:56.306163+0000","last_fullsized":"2024-03-24T10:33:56.306163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306298+0000","last_change":"2024-03-24T10:33:55.145385+0000","last_active":"2024-03-24T10:33:56.306298+0000","last_peered":"2024-03-24T10:33:56.306298+0000","last_clean":"2024-03-24T10:33:56.306298+0000","last_became_active":"2024-03-24T10:33:55.144766+0000","last_became_peered":"2024-03-24T10:33:55.144766+0000","last_unstale":"2024-03-24T10:33:56.306298+0000","last_undegraded":"2024-03-24T10:33:56.306298+0000","last_fullsized":"2024-03-24T10:33:56.306298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123213+0000","last_change":"2024-03-24T10:33:54.128955+0000","last_active":"2024-03-24T10:33:56.123213+0000","last_peered":"2024-03-24T10:33:56.123213+0000","last_clean":"2024-03-24T10:33:56.123213+0000","last_became_active":"2024-03-24T10:33:54.128654+0000","last_became_peered":"2024-03-24T10:33:54.128654+0000","last_unstale":"2024-03-24T10:33:56.123213+0000","last_undegraded":"2024-03-24T10:33:56.123213+0000","last_fullsized":"2024-03-24T10:33:56.123213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306400+0000","last_change":"2024-03-24T10:33:55.146595+0000","last_active":"2024-03-24T10:33:56.306400+0000","last_peered":"2024-03-24T10:33:56.306400+0000","last_clean":"2024-03-24T10:33:56.306400+0000","last_became_active":"2024-03-24T10:33:55.146309+0000","last_became_peered":"2024-03-24T10:33:55.146309+0000","last_unstale":"2024-03-24T10:33:56.306400+0000","last_undegraded":"2024-03-24T10:33:56.306400+0000","last_fullsized":"2024-03-24T10:33:56.306400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124069+0000","last_change":"2024-03-24T10:33:54.125637+0000","last_active":"2024-03-24T10:33:56.124069+0000","last_peered":"2024-03-24T10:33:56.124069+0000","last_clean":"2024-03-24T10:33:56.124069+0000","last_became_active":"2024-03-24T10:33:54.125278+0000","last_became_peered":"2024-03-24T10:33:54.125278+0000","last_unstale":"2024-03-24T10:33:56.124069+0000","last_undegraded":"2024-03-24T10:33:56.124069+0000","last_fullsized":"2024-03-24T10:33:56.124069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306668+0000","last_change":"2024-03-24T10:33:55.156480+0000","last_active":"2024-03-24T10:33:56.306668+0000","last_peered":"2024-03-24T10:33:56.306668+0000","last_clean":"2024-03-24T10:33:56.306668+0000","last_became_active":"2024-03-24T10:33:55.156201+0000","last_became_peered":"2024-03-24T10:33:55.156201+0000","last_unstale":"2024-03-24T10:33:56.306668+0000","last_undegraded":"2024-03-24T10:33:56.306668+0000","last_fullsized":"2024-03-24T10:33:56.306668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306091+0000","last_change":"2024-03-24T10:33:54.125505+0000","last_active":"2024-03-24T10:33:56.306091+0000","last_peered":"2024-03-24T10:33:56.306091+0000","last_clean":"2024-03-24T10:33:56.306091+0000","last_became_active":"2024-03-24T10:33:54.125170+0000","last_became_peered":"2024-03-24T10:33:54.125170+0000","last_unstale":"2024-03-24T10:33:56.306091+0000","last_undegraded":"2024-03-24T10:33:56.306091+0000","last_fullsized":"2024-03-24T10:33:56.306091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305748+0000","last_change":"2024-03-24T10:33:55.156955+0000","last_active":"2024-03-24T10:33:56.305748+0000","last_peered":"2024-03-24T10:33:56.305748+0000","last_clean":"2024-03-24T10:33:56.305748+0000","last_became_active":"2024-03-24T10:33:55.156565+0000","last_became_peered":"2024-03-24T10:33:55.156565+0000","last_unstale":"2024-03-24T10:33:56.305748+0000","last_undegraded":"2024-03-24T10:33:56.305748+0000","last_fullsized":"2024-03-24T10:33:56.305748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:32.795374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306167+0000","last_change":"2024-03-24T10:33:54.128904+0000","last_active":"2024-03-24T10:33:56.306167+0000","last_peered":"2024-03-24T10:33:56.306167+0000","last_clean":"2024-03-24T10:33:56.306167+0000","last_became_active":"2024-03-24T10:33:54.128349+0000","last_became_peered":"2024-03-24T10:33:54.128349+0000","last_unstale":"2024-03-24T10:33:56.306167+0000","last_undegraded":"2024-03-24T10:33:56.306167+0000","last_fullsized":"2024-03-24T10:33:56.306167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306695+0000","last_change":"2024-03-24T10:33:55.155945+0000","last_active":"2024-03-24T10:33:56.306695+0000","last_peered":"2024-03-24T10:33:56.306695+0000","last_clean":"2024-03-24T10:33:56.306695+0000","last_became_active":"2024-03-24T10:33:55.155660+0000","last_became_peered":"2024-03-24T10:33:55.155660+0000","last_unstale":"2024-03-24T10:33:56.306695+0000","last_undegraded":"2024-03-24T10:33:56.306695+0000","last_fullsized":"2024-03-24T10:33:56.306695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124002+0000","last_change":"2024-03-24T10:33:54.127975+0000","last_active":"2024-03-24T10:33:56.124002+0000","last_peered":"2024-03-24T10:33:56.124002+0000","last_clean":"2024-03-24T10:33:56.124002+0000","last_became_active":"2024-03-24T10:33:54.127705+0000","last_became_peered":"2024-03-24T10:33:54.127705+0000","last_unstale":"2024-03-24T10:33:56.124002+0000","last_undegraded":"2024-03-24T10:33:56.124002+0000","last_fullsized":"2024-03-24T10:33:56.124002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124453+0000","last_change":"2024-03-24T10:33:55.146773+0000","last_active":"2024-03-24T10:33:56.124453+0000","last_peered":"2024-03-24T10:33:56.124453+0000","last_clean":"2024-03-24T10:33:56.124453+0000","last_became_active":"2024-03-24T10:33:55.146086+0000","last_became_peered":"2024-03-24T10:33:55.146086+0000","last_unstale":"2024-03-24T10:33:56.124453+0000","last_undegraded":"2024-03-24T10:33:56.124453+0000","last_fullsized":"2024-03-24T10:33:56.124453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123547+0000","last_change":"2024-03-24T10:33:54.114004+0000","last_active":"2024-03-24T10:33:56.123547+0000","last_peered":"2024-03-24T10:33:56.123547+0000","last_clean":"2024-03-24T10:33:56.123547+0000","last_became_active":"2024-03-24T10:33:54.113714+0000","last_became_peered":"2024-03-24T10:33:54.113714+0000","last_unstale":"2024-03-24T10:33:56.123547+0000","last_undegraded":"2024-03-24T10:33:56.123547+0000","last_fullsized":"2024-03-24T10:33:56.123547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306338+0000","last_change":"2024-03-24T10:33:55.160713+0000","last_active":"2024-03-24T10:33:56.306338+0000","last_peered":"2024-03-24T10:33:56.306338+0000","last_clean":"2024-03-24T10:33:56.306338+0000","last_became_active":"2024-03-24T10:33:55.157835+0000","last_became_peered":"2024-03-24T10:33:55.157835+0000","last_unstale":"2024-03-24T10:33:56.306338+0000","last_undegraded":"2024-03-24T10:33:56.306338+0000","last_fullsized":"2024-03-24T10:33:56.306338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306046+0000","last_change":"2024-03-24T10:33:54.123641+0000","last_active":"2024-03-24T10:33:56.306046+0000","last_peered":"2024-03-24T10:33:56.306046+0000","last_clean":"2024-03-24T10:33:56.306046+0000","last_became_active":"2024-03-24T10:33:54.123045+0000","last_became_peered":"2024-03-24T10:33:54.123045+0000","last_unstale":"2024-03-24T10:33:56.306046+0000","last_undegraded":"2024-03-24T10:33:56.306046+0000","last_fullsized":"2024-03-24T10:33:56.306046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:41:41.902140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306617+0000","last_change":"2024-03-24T10:33:55.155307+0000","last_active":"2024-03-24T10:33:56.306617+0000","last_peered":"2024-03-24T10:33:56.306617+0000","last_clean":"2024-03-24T10:33:56.306617+0000","last_became_active":"2024-03-24T10:33:55.154976+0000","last_became_peered":"2024-03-24T10:33:55.154976+0000","last_unstale":"2024-03-24T10:33:56.306617+0000","last_undegraded":"2024-03-24T10:33:56.306617+0000","last_fullsized":"2024-03-24T10:33:56.306617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123767+0000","last_change":"2024-03-24T10:33:54.124359+0000","last_active":"2024-03-24T10:33:56.123767+0000","last_peered":"2024-03-24T10:33:56.123767+0000","last_clean":"2024-03-24T10:33:56.123767+0000","last_became_active":"2024-03-24T10:33:54.124063+0000","last_became_peered":"2024-03-24T10:33:54.124063+0000","last_unstale":"2024-03-24T10:33:56.123767+0000","last_undegraded":"2024-03-24T10:33:56.123767+0000","last_fullsized":"2024-03-24T10:33:56.123767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306899+0000","last_change":"2024-03-24T10:33:55.159546+0000","last_active":"2024-03-24T10:33:56.306899+0000","last_peered":"2024-03-24T10:33:56.306899+0000","last_clean":"2024-03-24T10:33:56.306899+0000","last_became_active":"2024-03-24T10:33:55.159193+0000","last_became_peered":"2024-03-24T10:33:55.159193+0000","last_unstale":"2024-03-24T10:33:56.306899+0000","last_undegraded":"2024-03-24T10:33:56.306899+0000","last_fullsized":"2024-03-24T10:33:56.306899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:07.558259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123249+0000","last_change":"2024-03-24T10:33:54.127205+0000","last_active":"2024-03-24T10:33:56.123249+0000","last_peered":"2024-03-24T10:33:56.123249+0000","last_clean":"2024-03-24T10:33:56.123249+0000","last_became_active":"2024-03-24T10:33:54.126938+0000","last_became_peered":"2024-03-24T10:33:54.126938+0000","last_unstale":"2024-03-24T10:33:56.123249+0000","last_undegraded":"2024-03-24T10:33:56.123249+0000","last_fullsized":"2024-03-24T10:33:56.123249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305970+0000","last_change":"2024-03-24T10:33:55.150130+0000","last_active":"2024-03-24T10:33:56.305970+0000","last_peered":"2024-03-24T10:33:56.305970+0000","last_clean":"2024-03-24T10:33:56.305970+0000","last_became_active":"2024-03-24T10:33:55.145689+0000","last_became_peered":"2024-03-24T10:33:55.145689+0000","last_unstale":"2024-03-24T10:33:56.305970+0000","last_undegraded":"2024-03-24T10:33:56.305970+0000","last_fullsized":"2024-03-24T10:33:56.305970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123759+0000","last_change":"2024-03-24T10:33:54.131439+0000","last_active":"2024-03-24T10:33:56.123759+0000","last_peered":"2024-03-24T10:33:56.123759+0000","last_clean":"2024-03-24T10:33:56.123759+0000","last_became_active":"2024-03-24T10:33:54.131117+0000","last_became_peered":"2024-03-24T10:33:54.131117+0000","last_unstale":"2024-03-24T10:33:56.123759+0000","last_undegraded":"2024-03-24T10:33:56.123759+0000","last_fullsized":"2024-03-24T10:33:56.123759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306791+0000","last_change":"2024-03-24T10:33:55.157088+0000","last_active":"2024-03-24T10:33:56.306791+0000","last_peered":"2024-03-24T10:33:56.306791+0000","last_clean":"2024-03-24T10:33:56.306791+0000","last_became_active":"2024-03-24T10:33:55.156790+0000","last_became_peered":"2024-03-24T10:33:55.156790+0000","last_unstale":"2024-03-24T10:33:56.306791+0000","last_undegraded":"2024-03-24T10:33:56.306791+0000","last_fullsized":"2024-03-24T10:33:56.306791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306529+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:33:56.306529+0000","last_peered":"2024-03-24T10:33:56.306529+0000","last_clean":"2024-03-24T10:33:56.306529+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:33:56.306529+0000","last_undegraded":"2024-03-24T10:33:56.306529+0000","last_fullsized":"2024-03-24T10:33:56.306529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305276+0000","last_change":"2024-03-24T10:33:54.121820+0000","last_active":"2024-03-24T10:33:56.305276+0000","last_peered":"2024-03-24T10:33:56.305276+0000","last_clean":"2024-03-24T10:33:56.305276+0000","last_became_active":"2024-03-24T10:33:54.121496+0000","last_became_peered":"2024-03-24T10:33:54.121496+0000","last_unstale":"2024-03-24T10:33:56.305276+0000","last_undegraded":"2024-03-24T10:33:56.305276+0000","last_fullsized":"2024-03-24T10:33:56.305276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305400+0000","last_change":"2024-03-24T10:33:55.146810+0000","last_active":"2024-03-24T10:33:56.305400+0000","last_peered":"2024-03-24T10:33:56.305400+0000","last_clean":"2024-03-24T10:33:56.305400+0000","last_became_active":"2024-03-24T10:33:55.146508+0000","last_became_peered":"2024-03-24T10:33:55.146508+0000","last_unstale":"2024-03-24T10:33:56.305400+0000","last_undegraded":"2024-03-24T10:33:56.305400+0000","last_fullsized":"2024-03-24T10:33:56.305400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306540+0000","last_change":"2024-03-24T10:33:54.127727+0000","last_active":"2024-03-24T10:33:56.306540+0000","last_peered":"2024-03-24T10:33:56.306540+0000","last_clean":"2024-03-24T10:33:56.306540+0000","last_became_active":"2024-03-24T10:33:54.127106+0000","last_became_peered":"2024-03-24T10:33:54.127106+0000","last_unstale":"2024-03-24T10:33:56.306540+0000","last_undegraded":"2024-03-24T10:33:56.306540+0000","last_fullsized":"2024-03-24T10:33:56.306540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:31.791505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306196+0000","last_change":"2024-03-24T10:33:55.156425+0000","last_active":"2024-03-24T10:33:56.306196+0000","last_peered":"2024-03-24T10:33:56.306196+0000","last_clean":"2024-03-24T10:33:56.306196+0000","last_became_active":"2024-03-24T10:33:55.156146+0000","last_became_peered":"2024-03-24T10:33:55.156146+0000","last_unstale":"2024-03-24T10:33:56.306196+0000","last_undegraded":"2024-03-24T10:33:56.306196+0000","last_fullsized":"2024-03-24T10:33:56.306196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123643+0000","last_change":"2024-03-24T10:33:54.127887+0000","last_active":"2024-03-24T10:33:56.123643+0000","last_peered":"2024-03-24T10:33:56.123643+0000","last_clean":"2024-03-24T10:33:56.123643+0000","last_became_active":"2024-03-24T10:33:54.127602+0000","last_became_peered":"2024-03-24T10:33:54.127602+0000","last_unstale":"2024-03-24T10:33:56.123643+0000","last_undegraded":"2024-03-24T10:33:56.123643+0000","last_fullsized":"2024-03-24T10:33:56.123643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122824+0000","last_change":"2024-03-24T10:33:55.128811+0000","last_active":"2024-03-24T10:33:56.122824+0000","last_peered":"2024-03-24T10:33:56.122824+0000","last_clean":"2024-03-24T10:33:56.122824+0000","last_became_active":"2024-03-24T10:33:55.128094+0000","last_became_peered":"2024-03-24T10:33:55.128094+0000","last_unstale":"2024-03-24T10:33:56.122824+0000","last_undegraded":"2024-03-24T10:33:56.122824+0000","last_fullsized":"2024-03-24T10:33:56.122824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306104+0000","last_change":"2024-03-24T10:33:54.123812+0000","last_active":"2024-03-24T10:33:56.306104+0000","last_peered":"2024-03-24T10:33:56.306104+0000","last_clean":"2024-03-24T10:33:56.306104+0000","last_became_active":"2024-03-24T10:33:54.122614+0000","last_became_peered":"2024-03-24T10:33:54.122614+0000","last_unstale":"2024-03-24T10:33:56.306104+0000","last_undegraded":"2024-03-24T10:33:56.306104+0000","last_fullsized":"2024-03-24T10:33:56.306104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305946+0000","last_change":"2024-03-24T10:33:55.140461+0000","last_active":"2024-03-24T10:33:56.305946+0000","last_peered":"2024-03-24T10:33:56.305946+0000","last_clean":"2024-03-24T10:33:56.305946+0000","last_became_active":"2024-03-24T10:33:55.139737+0000","last_became_peered":"2024-03-24T10:33:55.139737+0000","last_unstale":"2024-03-24T10:33:56.305946+0000","last_undegraded":"2024-03-24T10:33:56.305946+0000","last_fullsized":"2024-03-24T10:33:56.305946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:27.228493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122944+0000","last_change":"2024-03-24T10:33:54.123506+0000","last_active":"2024-03-24T10:33:56.122944+0000","last_peered":"2024-03-24T10:33:56.122944+0000","last_clean":"2024-03-24T10:33:56.122944+0000","last_became_active":"2024-03-24T10:33:54.123194+0000","last_became_peered":"2024-03-24T10:33:54.123194+0000","last_unstale":"2024-03-24T10:33:56.122944+0000","last_undegraded":"2024-03-24T10:33:56.122944+0000","last_fullsized":"2024-03-24T10:33:56.122944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305900+0000","last_change":"2024-03-24T10:33:55.148696+0000","last_active":"2024-03-24T10:33:56.305900+0000","last_peered":"2024-03-24T10:33:56.305900+0000","last_clean":"2024-03-24T10:33:56.305900+0000","last_became_active":"2024-03-24T10:33:55.148382+0000","last_became_peered":"2024-03-24T10:33:55.148382+0000","last_unstale":"2024-03-24T10:33:56.305900+0000","last_undegraded":"2024-03-24T10:33:56.305900+0000","last_fullsized":"2024-03-24T10:33:56.305900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306970+0000","last_change":"2024-03-24T10:33:54.129296+0000","last_active":"2024-03-24T10:33:56.306970+0000","last_peered":"2024-03-24T10:33:56.306970+0000","last_clean":"2024-03-24T10:33:56.306970+0000","last_became_active":"2024-03-24T10:33:54.128788+0000","last_became_peered":"2024-03-24T10:33:54.128788+0000","last_unstale":"2024-03-24T10:33:56.306970+0000","last_undegraded":"2024-03-24T10:33:56.306970+0000","last_fullsized":"2024-03-24T10:33:56.306970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:48.412131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122832+0000","last_change":"2024-03-24T10:33:55.128461+0000","last_active":"2024-03-24T10:33:56.122832+0000","last_peered":"2024-03-24T10:33:56.122832+0000","last_clean":"2024-03-24T10:33:56.122832+0000","last_became_active":"2024-03-24T10:33:55.128125+0000","last_became_peered":"2024-03-24T10:33:55.128125+0000","last_unstale":"2024-03-24T10:33:56.122832+0000","last_undegraded":"2024-03-24T10:33:56.122832+0000","last_fullsized":"2024-03-24T10:33:56.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.140759+0000","last_change":"2024-03-24T10:33:54.117337+0000","last_active":"2024-03-24T10:33:56.140759+0000","last_peered":"2024-03-24T10:33:56.140759+0000","last_clean":"2024-03-24T10:33:56.140759+0000","last_became_active":"2024-03-24T10:33:54.116814+0000","last_became_peered":"2024-03-24T10:33:54.116814+0000","last_unstale":"2024-03-24T10:33:56.140759+0000","last_undegraded":"2024-03-24T10:33:56.140759+0000","last_fullsized":"2024-03-24T10:33:56.140759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.147551+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.147261+0000","last_became_peered":"2024-03-24T10:33:55.147261+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124432+0000","last_change":"2024-03-24T10:33:54.127434+0000","last_active":"2024-03-24T10:33:56.124432+0000","last_peered":"2024-03-24T10:33:56.124432+0000","last_clean":"2024-03-24T10:33:56.124432+0000","last_became_active":"2024-03-24T10:33:54.127162+0000","last_became_peered":"2024-03-24T10:33:54.127162+0000","last_unstale":"2024-03-24T10:33:56.124432+0000","last_undegraded":"2024-03-24T10:33:56.124432+0000","last_fullsized":"2024-03-24T10:33:56.124432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306039+0000","last_change":"2024-03-24T10:33:55.142227+0000","last_active":"2024-03-24T10:33:56.306039+0000","last_peered":"2024-03-24T10:33:56.306039+0000","last_clean":"2024-03-24T10:33:56.306039+0000","last_became_active":"2024-03-24T10:33:55.141920+0000","last_became_peered":"2024-03-24T10:33:55.141920+0000","last_unstale":"2024-03-24T10:33:56.306039+0000","last_undegraded":"2024-03-24T10:33:56.306039+0000","last_fullsized":"2024-03-24T10:33:56.306039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306452+0000","last_change":"2024-03-24T10:33:54.128286+0000","last_active":"2024-03-24T10:33:56.306452+0000","last_peered":"2024-03-24T10:33:56.306452+0000","last_clean":"2024-03-24T10:33:56.306452+0000","last_became_active":"2024-03-24T10:33:54.127434+0000","last_became_peered":"2024-03-24T10:33:54.127434+0000","last_unstale":"2024-03-24T10:33:56.306452+0000","last_undegraded":"2024-03-24T10:33:56.306452+0000","last_fullsized":"2024-03-24T10:33:56.306452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:40.602694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305662+0000","last_change":"2024-03-24T10:33:55.132582+0000","last_active":"2024-03-24T10:33:56.305662+0000","last_peered":"2024-03-24T10:33:56.305662+0000","last_clean":"2024-03-24T10:33:56.305662+0000","last_became_active":"2024-03-24T10:33:55.132259+0000","last_became_peered":"2024-03-24T10:33:55.132259+0000","last_unstale":"2024-03-24T10:33:56.305662+0000","last_undegraded":"2024-03-24T10:33:56.305662+0000","last_fullsized":"2024-03-24T10:33:56.305662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:22.596121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122894+0000","last_change":"2024-03-24T10:33:54.127899+0000","last_active":"2024-03-24T10:33:56.122894+0000","last_peered":"2024-03-24T10:33:56.122894+0000","last_clean":"2024-03-24T10:33:56.122894+0000","last_became_active":"2024-03-24T10:33:54.127592+0000","last_became_peered":"2024-03-24T10:33:54.127592+0000","last_unstale":"2024-03-24T10:33:56.122894+0000","last_undegraded":"2024-03-24T10:33:56.122894+0000","last_fullsized":"2024-03-24T10:33:56.122894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305592+0000","last_change":"2024-03-24T10:33:55.133853+0000","last_active":"2024-03-24T10:33:56.305592+0000","last_peered":"2024-03-24T10:33:56.305592+0000","last_clean":"2024-03-24T10:33:56.305592+0000","last_became_active":"2024-03-24T10:33:55.133478+0000","last_became_peered":"2024-03-24T10:33:55.133478+0000","last_unstale":"2024-03-24T10:33:56.305592+0000","last_undegraded":"2024-03-24T10:33:56.305592+0000","last_fullsized":"2024-03-24T10:33:56.305592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123001+0000","last_change":"2024-03-24T10:33:54.126665+0000","last_active":"2024-03-24T10:33:56.123001+0000","last_peered":"2024-03-24T10:33:56.123001+0000","last_clean":"2024-03-24T10:33:56.123001+0000","last_became_active":"2024-03-24T10:33:54.126339+0000","last_became_peered":"2024-03-24T10:33:54.126339+0000","last_unstale":"2024-03-24T10:33:56.123001+0000","last_undegraded":"2024-03-24T10:33:56.123001+0000","last_fullsized":"2024-03-24T10:33:56.123001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305669+0000","last_change":"2024-03-24T10:33:55.140585+0000","last_active":"2024-03-24T10:33:56.305669+0000","last_peered":"2024-03-24T10:33:56.305669+0000","last_clean":"2024-03-24T10:33:56.305669+0000","last_became_active":"2024-03-24T10:33:55.140142+0000","last_became_peered":"2024-03-24T10:33:55.140142+0000","last_unstale":"2024-03-24T10:33:56.305669+0000","last_undegraded":"2024-03-24T10:33:56.305669+0000","last_fullsized":"2024-03-24T10:33:56.305669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306757+0000","last_change":"2024-03-24T10:33:54.128102+0000","last_active":"2024-03-24T10:33:56.306757+0000","last_peered":"2024-03-24T10:33:56.306757+0000","last_clean":"2024-03-24T10:33:56.306757+0000","last_became_active":"2024-03-24T10:33:54.127077+0000","last_became_peered":"2024-03-24T10:33:54.127077+0000","last_unstale":"2024-03-24T10:33:56.306757+0000","last_undegraded":"2024-03-24T10:33:56.306757+0000","last_fullsized":"2024-03-24T10:33:56.306757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123921+0000","last_change":"2024-03-24T10:33:55.155558+0000","last_active":"2024-03-24T10:33:56.123921+0000","last_peered":"2024-03-24T10:33:56.123921+0000","last_clean":"2024-03-24T10:33:56.123921+0000","last_became_active":"2024-03-24T10:33:55.155291+0000","last_became_peered":"2024-03-24T10:33:55.155291+0000","last_unstale":"2024-03-24T10:33:56.123921+0000","last_undegraded":"2024-03-24T10:33:56.123921+0000","last_fullsized":"2024-03-24T10:33:56.123921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305743+0000","last_change":"2024-03-24T10:33:55.159176+0000","last_active":"2024-03-24T10:33:56.305743+0000","last_peered":"2024-03-24T10:33:56.305743+0000","last_clean":"2024-03-24T10:33:56.305743+0000","last_became_active":"2024-03-24T10:33:55.158885+0000","last_became_peered":"2024-03-24T10:33:55.158885+0000","last_unstale":"2024-03-24T10:33:56.305743+0000","last_undegraded":"2024-03-24T10:33:56.305743+0000","last_fullsized":"2024-03-24T10:33:56.305743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306353+0000","last_change":"2024-03-24T10:33:54.132298+0000","last_active":"2024-03-24T10:33:56.306353+0000","last_peered":"2024-03-24T10:33:56.306353+0000","last_clean":"2024-03-24T10:33:56.306353+0000","last_became_active":"2024-03-24T10:33:54.132009+0000","last_became_peered":"2024-03-24T10:33:54.132009+0000","last_unstale":"2024-03-24T10:33:56.306353+0000","last_undegraded":"2024-03-24T10:33:56.306353+0000","last_fullsized":"2024-03-24T10:33:56.306353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305825+0000","last_change":"2024-03-24T10:33:55.159162+0000","last_active":"2024-03-24T10:33:56.305825+0000","last_peered":"2024-03-24T10:33:56.305825+0000","last_clean":"2024-03-24T10:33:56.305825+0000","last_became_active":"2024-03-24T10:33:55.158898+0000","last_became_peered":"2024-03-24T10:33:55.158898+0000","last_unstale":"2024-03-24T10:33:56.305825+0000","last_undegraded":"2024-03-24T10:33:56.305825+0000","last_fullsized":"2024-03-24T10:33:56.305825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306863+0000","last_change":"2024-03-24T10:33:54.129068+0000","last_active":"2024-03-24T10:33:56.306863+0000","last_peered":"2024-03-24T10:33:56.306863+0000","last_clean":"2024-03-24T10:33:56.306863+0000","last_became_active":"2024-03-24T10:33:54.128465+0000","last_became_peered":"2024-03-24T10:33:54.128465+0000","last_unstale":"2024-03-24T10:33:56.306863+0000","last_undegraded":"2024-03-24T10:33:56.306863+0000","last_fullsized":"2024-03-24T10:33:56.306863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306248+0000","last_change":"2024-03-24T10:33:55.146210+0000","last_active":"2024-03-24T10:33:56.306248+0000","last_peered":"2024-03-24T10:33:56.306248+0000","last_clean":"2024-03-24T10:33:56.306248+0000","last_became_active":"2024-03-24T10:33:55.145069+0000","last_became_peered":"2024-03-24T10:33:55.145069+0000","last_unstale":"2024-03-24T10:33:56.306248+0000","last_undegraded":"2024-03-24T10:33:56.306248+0000","last_fullsized":"2024-03-24T10:33:56.306248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305927+0000","last_change":"2024-03-24T10:33:54.126679+0000","last_active":"2024-03-24T10:33:56.305927+0000","last_peered":"2024-03-24T10:33:56.305927+0000","last_clean":"2024-03-24T10:33:56.305927+0000","last_became_active":"2024-03-24T10:33:54.126407+0000","last_became_peered":"2024-03-24T10:33:54.126407+0000","last_unstale":"2024-03-24T10:33:56.305927+0000","last_undegraded":"2024-03-24T10:33:56.305927+0000","last_fullsized":"2024-03-24T10:33:56.305927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305615+0000","last_change":"2024-03-24T10:33:55.150245+0000","last_active":"2024-03-24T10:33:56.305615+0000","last_peered":"2024-03-24T10:33:56.305615+0000","last_clean":"2024-03-24T10:33:56.305615+0000","last_became_active":"2024-03-24T10:33:55.149927+0000","last_became_peered":"2024-03-24T10:33:55.149927+0000","last_unstale":"2024-03-24T10:33:56.305615+0000","last_undegraded":"2024-03-24T10:33:56.305615+0000","last_fullsized":"2024-03-24T10:33:56.305615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306337+0000","last_change":"2024-03-24T10:33:54.127624+0000","last_active":"2024-03-24T10:33:56.306337+0000","last_peered":"2024-03-24T10:33:56.306337+0000","last_clean":"2024-03-24T10:33:56.306337+0000","last_became_active":"2024-03-24T10:33:54.127307+0000","last_became_peered":"2024-03-24T10:33:54.127307+0000","last_unstale":"2024-03-24T10:33:56.306337+0000","last_undegraded":"2024-03-24T10:33:56.306337+0000","last_fullsized":"2024-03-24T10:33:56.306337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:33:59.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:33:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.266+0000 7efd0aa8d640 1 Processor -- start 2024-03-24T10:33:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.266+0000 7efd0aa8d640 1 -- start start 2024-03-24T10:33:59.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0aa8d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd0415e390 0x7efd040d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0aa8d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040b4ba0 0x7efd040d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0aa8d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0415e770 con 0x7efd040d1000 2024-03-24T10:33:59.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0aa8d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efd040b3190 con 0x7efd0415e390 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0aa8d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efd040d4e40 con 0x7efd040b4ba0 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd08829640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd0415e390 0x7efd040d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd08829640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd0415e390 0x7efd040d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45382/0 (socket says 172.21.15.1:45382) 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd03fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efd040d1000 0x7efd040cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39598/0 (socket says 172.21.15.1:39598) 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd08829640 1 -- 172.21.15.1:0/2001667088 learned_addr learned my addr 172.21.15.1:0/2001667088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.267+0000 7efd0902a640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040b4ba0 0x7efd040d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd08829640 1 -- 172.21.15.1:0/2001667088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040b4ba0 msgr2=0x7efd040d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd08829640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040b4ba0 0x7efd040d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd08829640 1 -- 172.21.15.1:0/2001667088 >> v1:172.21.15.1:6789/0 conn(0x7efd040d1000 legacy=0x7efd040cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd08829640 1 -- 172.21.15.1:0/2001667088 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040d1e40 con 0x7efd0415e390 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd08829640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd0415e390 0x7efd040d24a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009870 tx=0x7efcf802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7434175c3f2c539 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd037fe640 1 -- 172.21.15.1:0/2001667088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80317c0 con 0x7efd0415e390 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.268+0000 7efd037fe640 1 -- 172.21.15.1:0/2001667088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8031920 con 0x7efd0415e390 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd037fe640 1 -- 172.21.15.1:0/2001667088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8031cb0 con 0x7efd0415e390 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 -- 172.21.15.1:0/2001667088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd0415e390 msgr2=0x7efd040d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd0415e390 0x7efd040d24a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009870 tx=0x7efcf802b860 comp rx=0 tx=0).stop 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 -- 172.21.15.1:0/2001667088 shutdown_connections 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040b4ba0 0x7efd040d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/2001667088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd0415e390 0x7efd040d24a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 -- 172.21.15.1:0/2001667088 >> 172.21.15.1:0/2001667088 conn(0x7efd040d3ec0 msgr2=0x7efd040d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 -- 172.21.15.1:0/2001667088 shutdown_connections 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.269+0000 7efd0aa8d640 1 -- 172.21.15.1:0/2001667088 wait complete. 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.270+0000 7efd0aa8d640 1 Processor -- start 2024-03-24T10:33:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.270+0000 7efd0aa8d640 1 -- start start 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040d1000 0x7efd040b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0415e390 0x7efd040cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efd0414f850 con 0x7efd040b4ba0 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efd040b8610 con 0x7efd0415e390 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0aa8d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efd040b88f0 con 0x7efd040d1000 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45388/0 (socket says 172.21.15.1:45388) 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 -- 172.21.15.1:0/512086349 learned_addr learned my addr 172.21.15.1:0/512086349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd0902a640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0415e390 0x7efd040cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd03fff640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040d1000 0x7efd040b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 -- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040d1000 msgr2=0x7efd040b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040d1000 0x7efd040b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 -- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0415e390 msgr2=0x7efd040cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0415e390 0x7efd040cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.271+0000 7efd08829640 1 -- 172.21.15.1:0/512086349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf80094d0 con 0x7efd040b4ba0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd08829640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efcf802b8a0 tx=0x7efcf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8032860 con 0x7efd040b4ba0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8032a00 con 0x7efd040b4ba0 2024-03-24T10:33:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd040b8bd0 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8032db0 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.272+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd040b9190 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.273+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd040b93b0 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.273+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efcf804e450 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.274+0000 7efd017fa640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 0x7efce806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.274+0000 7efd03fff640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 0x7efce806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.274+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efcf80ac0c0 con 0x7efd040b4ba0 2024-03-24T10:33:59.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.274+0000 7efd03fff640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 0x7efce806dd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd040a1de0 tx=0x7efcf0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.278+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcf8076c20 con 0x7efd040b4ba0 2024-03-24T10:33:59.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.455+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd040b3320 con 0x7efce806b840 2024-03-24T10:33:59.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.461+0000 7efd017fa640 1 -- 172.21.15.1:0/512086349 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7efcf0004aa0 con 0x7efce806b840 2024-03-24T10:33:59.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 msgr2=0x7efce806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 0x7efce806dd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd040a1de0 tx=0x7efcf0007e80 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 msgr2=0x7efd0414f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efcf802b8a0 tx=0x7efcf80046c0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 shutdown_connections 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efce806b840 0x7efce806dd00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd0415e390 0x7efd040cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efd040d1000 0x7efd040b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 --2- 172.21.15.1:0/512086349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040b4ba0 0x7efd0414f310 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 >> 172.21.15.1:0/512086349 conn(0x7efd040d3ec0 msgr2=0x7efd040d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 shutdown_connections 2024-03-24T10:33:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.464+0000 7efd0aa8d640 1 -- 172.21.15.1:0/512086349 wait complete. 2024-03-24T10:33:59.475 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-24T10:33:58.517982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235656,"kb_used_data":2848,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739064,"statfs":{"total":773094113280,"available":772852801536,"internally_reserved":0,"allocated":2916352,"data_stored":1752596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412770"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306747+0000","last_change":"2024-03-24T10:33:55.160859+0000","last_active":"2024-03-24T10:33:56.306747+0000","last_peered":"2024-03-24T10:33:56.306747+0000","last_clean":"2024-03-24T10:33:56.306747+0000","last_became_active":"2024-03-24T10:33:55.160069+0000","last_became_peered":"2024-03-24T10:33:55.160069+0000","last_unstale":"2024-03-24T10:33:56.306747+0000","last_undegraded":"2024-03-24T10:33:56.306747+0000","last_fullsized":"2024-03-24T10:33:56.306747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123102+0000","last_change":"2024-03-24T10:33:55.140972+0000","last_active":"2024-03-24T10:33:56.123102+0000","last_peered":"2024-03-24T10:33:56.123102+0000","last_clean":"2024-03-24T10:33:56.123102+0000","last_became_active":"2024-03-24T10:33:55.140573+0000","last_became_peered":"2024-03-24T10:33:55.140573+0000","last_unstale":"2024-03-24T10:33:56.123102+0000","last_undegraded":"2024-03-24T10:33:56.123102+0000","last_fullsized":"2024-03-24T10:33:56.123102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305754+0000","last_change":"2024-03-24T10:33:55.141078+0000","last_active":"2024-03-24T10:33:56.305754+0000","last_peered":"2024-03-24T10:33:56.305754+0000","last_clean":"2024-03-24T10:33:56.305754+0000","last_became_active":"2024-03-24T10:33:55.140774+0000","last_became_peered":"2024-03-24T10:33:55.140774+0000","last_unstale":"2024-03-24T10:33:56.305754+0000","last_undegraded":"2024-03-24T10:33:56.305754+0000","last_fullsized":"2024-03-24T10:33:56.305754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305846+0000","last_change":"2024-03-24T10:33:55.148019+0000","last_active":"2024-03-24T10:33:56.305846+0000","last_peered":"2024-03-24T10:33:56.305846+0000","last_clean":"2024-03-24T10:33:56.305846+0000","last_became_active":"2024-03-24T10:33:55.145313+0000","last_became_peered":"2024-03-24T10:33:55.145313+0000","last_unstale":"2024-03-24T10:33:56.305846+0000","last_undegraded":"2024-03-24T10:33:56.305846+0000","last_fullsized":"2024-03-24T10:33:56.305846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306869+0000","last_change":"2024-03-24T10:33:55.145334+0000","last_active":"2024-03-24T10:33:56.306869+0000","last_peered":"2024-03-24T10:33:56.306869+0000","last_clean":"2024-03-24T10:33:56.306869+0000","last_became_active":"2024-03-24T10:33:55.145018+0000","last_became_peered":"2024-03-24T10:33:55.145018+0000","last_unstale":"2024-03-24T10:33:56.306869+0000","last_undegraded":"2024-03-24T10:33:56.306869+0000","last_fullsized":"2024-03-24T10:33:56.306869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:24:24.292025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305901+0000","last_change":"2024-03-24T10:33:55.142392+0000","last_active":"2024-03-24T10:33:56.305901+0000","last_peered":"2024-03-24T10:33:56.305901+0000","last_clean":"2024-03-24T10:33:56.305901+0000","last_became_active":"2024-03-24T10:33:55.142070+0000","last_became_peered":"2024-03-24T10:33:55.142070+0000","last_unstale":"2024-03-24T10:33:56.305901+0000","last_undegraded":"2024-03-24T10:33:56.305901+0000","last_fullsized":"2024-03-24T10:33:56.305901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306206+0000","last_change":"2024-03-24T10:33:55.158958+0000","last_active":"2024-03-24T10:33:56.306206+0000","last_peered":"2024-03-24T10:33:56.306206+0000","last_clean":"2024-03-24T10:33:56.306206+0000","last_became_active":"2024-03-24T10:33:55.158270+0000","last_became_peered":"2024-03-24T10:33:55.158270+0000","last_unstale":"2024-03-24T10:33:56.306206+0000","last_undegraded":"2024-03-24T10:33:56.306206+0000","last_fullsized":"2024-03-24T10:33:56.306206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305233+0000","last_change":"2024-03-24T10:33:55.142337+0000","last_active":"2024-03-24T10:33:56.305233+0000","last_peered":"2024-03-24T10:33:56.305233+0000","last_clean":"2024-03-24T10:33:56.305233+0000","last_became_active":"2024-03-24T10:33:55.141994+0000","last_became_peered":"2024-03-24T10:33:55.141994+0000","last_unstale":"2024-03-24T10:33:56.305233+0000","last_undegraded":"2024-03-24T10:33:56.305233+0000","last_fullsized":"2024-03-24T10:33:56.305233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306802+0000","last_change":"2024-03-24T10:33:55.155886+0000","last_active":"2024-03-24T10:33:56.306802+0000","last_peered":"2024-03-24T10:33:56.306802+0000","last_clean":"2024-03-24T10:33:56.306802+0000","last_became_active":"2024-03-24T10:33:55.153792+0000","last_became_peered":"2024-03-24T10:33:55.153792+0000","last_unstale":"2024-03-24T10:33:56.306802+0000","last_undegraded":"2024-03-24T10:33:56.306802+0000","last_fullsized":"2024-03-24T10:33:56.306802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124307+0000","last_change":"2024-03-24T10:33:55.152791+0000","last_active":"2024-03-24T10:33:56.124307+0000","last_peered":"2024-03-24T10:33:56.124307+0000","last_clean":"2024-03-24T10:33:56.124307+0000","last_became_active":"2024-03-24T10:33:55.152524+0000","last_became_peered":"2024-03-24T10:33:55.152524+0000","last_unstale":"2024-03-24T10:33:56.124307+0000","last_undegraded":"2024-03-24T10:33:56.124307+0000","last_fullsized":"2024-03-24T10:33:56.124307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124236+0000","last_change":"2024-03-24T10:33:55.153894+0000","last_active":"2024-03-24T10:33:56.124236+0000","last_peered":"2024-03-24T10:33:56.124236+0000","last_clean":"2024-03-24T10:33:56.124236+0000","last_became_active":"2024-03-24T10:33:55.153623+0000","last_became_peered":"2024-03-24T10:33:55.153623+0000","last_unstale":"2024-03-24T10:33:56.124236+0000","last_undegraded":"2024-03-24T10:33:56.124236+0000","last_fullsized":"2024-03-24T10:33:56.124236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306325+0000","last_change":"2024-03-24T10:33:55.148337+0000","last_active":"2024-03-24T10:33:56.306325+0000","last_peered":"2024-03-24T10:33:56.306325+0000","last_clean":"2024-03-24T10:33:56.306325+0000","last_became_active":"2024-03-24T10:33:55.148076+0000","last_became_peered":"2024-03-24T10:33:55.148076+0000","last_unstale":"2024-03-24T10:33:56.306325+0000","last_undegraded":"2024-03-24T10:33:56.306325+0000","last_fullsized":"2024-03-24T10:33:56.306325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305500+0000","last_change":"2024-03-24T10:33:55.148625+0000","last_active":"2024-03-24T10:33:56.305500+0000","last_peered":"2024-03-24T10:33:56.305500+0000","last_clean":"2024-03-24T10:33:56.305500+0000","last_became_active":"2024-03-24T10:33:55.148234+0000","last_became_peered":"2024-03-24T10:33:55.148234+0000","last_unstale":"2024-03-24T10:33:56.305500+0000","last_undegraded":"2024-03-24T10:33:56.305500+0000","last_fullsized":"2024-03-24T10:33:56.305500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305594+0000","last_change":"2024-03-24T10:33:55.141825+0000","last_active":"2024-03-24T10:33:56.305594+0000","last_peered":"2024-03-24T10:33:56.305594+0000","last_clean":"2024-03-24T10:33:56.305594+0000","last_became_active":"2024-03-24T10:33:55.141380+0000","last_became_peered":"2024-03-24T10:33:55.141380+0000","last_unstale":"2024-03-24T10:33:56.305594+0000","last_undegraded":"2024-03-24T10:33:56.305594+0000","last_fullsized":"2024-03-24T10:33:56.305594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:51.274453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122775+0000","last_change":"2024-03-24T10:33:55.132926+0000","last_active":"2024-03-24T10:33:56.122775+0000","last_peered":"2024-03-24T10:33:56.122775+0000","last_clean":"2024-03-24T10:33:56.122775+0000","last_became_active":"2024-03-24T10:33:55.132614+0000","last_became_peered":"2024-03-24T10:33:55.132614+0000","last_unstale":"2024-03-24T10:33:56.122775+0000","last_undegraded":"2024-03-24T10:33:56.122775+0000","last_fullsized":"2024-03-24T10:33:56.122775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305340+0000","last_change":"2024-03-24T10:33:55.141440+0000","last_active":"2024-03-24T10:33:56.305340+0000","last_peered":"2024-03-24T10:33:56.305340+0000","last_clean":"2024-03-24T10:33:56.305340+0000","last_became_active":"2024-03-24T10:33:55.141096+0000","last_became_peered":"2024-03-24T10:33:55.141096+0000","last_unstale":"2024-03-24T10:33:56.305340+0000","last_undegraded":"2024-03-24T10:33:56.305340+0000","last_fullsized":"2024-03-24T10:33:56.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306420+0000","last_change":"2024-03-24T10:33:55.152651+0000","last_active":"2024-03-24T10:33:56.306420+0000","last_peered":"2024-03-24T10:33:56.306420+0000","last_clean":"2024-03-24T10:33:56.306420+0000","last_became_active":"2024-03-24T10:33:55.152386+0000","last_became_peered":"2024-03-24T10:33:55.152386+0000","last_unstale":"2024-03-24T10:33:56.306420+0000","last_undegraded":"2024-03-24T10:33:56.306420+0000","last_fullsized":"2024-03-24T10:33:56.306420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123835+0000","last_change":"2024-03-24T10:33:55.149184+0000","last_active":"2024-03-24T10:33:56.123835+0000","last_peered":"2024-03-24T10:33:56.123835+0000","last_clean":"2024-03-24T10:33:56.123835+0000","last_became_active":"2024-03-24T10:33:55.148828+0000","last_became_peered":"2024-03-24T10:33:55.148828+0000","last_unstale":"2024-03-24T10:33:56.123835+0000","last_undegraded":"2024-03-24T10:33:56.123835+0000","last_fullsized":"2024-03-24T10:33:56.123835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:33.564193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306947+0000","last_change":"2024-03-24T10:33:55.147090+0000","last_active":"2024-03-24T10:33:56.306947+0000","last_peered":"2024-03-24T10:33:56.306947+0000","last_clean":"2024-03-24T10:33:56.306947+0000","last_became_active":"2024-03-24T10:33:55.146820+0000","last_became_peered":"2024-03-24T10:33:55.146820+0000","last_unstale":"2024-03-24T10:33:56.306947+0000","last_undegraded":"2024-03-24T10:33:56.306947+0000","last_fullsized":"2024-03-24T10:33:56.306947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:12.251747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124127+0000","last_change":"2024-03-24T10:33:55.156434+0000","last_active":"2024-03-24T10:33:56.124127+0000","last_peered":"2024-03-24T10:33:56.124127+0000","last_clean":"2024-03-24T10:33:56.124127+0000","last_became_active":"2024-03-24T10:33:55.156176+0000","last_became_peered":"2024-03-24T10:33:55.156176+0000","last_unstale":"2024-03-24T10:33:56.124127+0000","last_undegraded":"2024-03-24T10:33:56.124127+0000","last_fullsized":"2024-03-24T10:33:56.124127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305688+0000","last_change":"2024-03-24T10:33:55.155986+0000","last_active":"2024-03-24T10:33:56.305688+0000","last_peered":"2024-03-24T10:33:56.305688+0000","last_clean":"2024-03-24T10:33:56.305688+0000","last_became_active":"2024-03-24T10:33:55.155699+0000","last_became_peered":"2024-03-24T10:33:55.155699+0000","last_unstale":"2024-03-24T10:33:56.305688+0000","last_undegraded":"2024-03-24T10:33:56.305688+0000","last_fullsized":"2024-03-24T10:33:56.305688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:41.606563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305535+0000","last_change":"2024-03-24T10:33:55.151865+0000","last_active":"2024-03-24T10:33:56.305535+0000","last_peered":"2024-03-24T10:33:56.305535+0000","last_clean":"2024-03-24T10:33:56.305535+0000","last_became_active":"2024-03-24T10:33:55.151592+0000","last_became_peered":"2024-03-24T10:33:55.151592+0000","last_unstale":"2024-03-24T10:33:56.305535+0000","last_undegraded":"2024-03-24T10:33:56.305535+0000","last_fullsized":"2024-03-24T10:33:56.305535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305309+0000","last_change":"2024-03-24T10:33:55.152238+0000","last_active":"2024-03-24T10:33:56.305309+0000","last_peered":"2024-03-24T10:33:56.305309+0000","last_clean":"2024-03-24T10:33:56.305309+0000","last_became_active":"2024-03-24T10:33:55.151966+0000","last_became_peered":"2024-03-24T10:33:55.151966+0000","last_unstale":"2024-03-24T10:33:56.305309+0000","last_undegraded":"2024-03-24T10:33:56.305309+0000","last_fullsized":"2024-03-24T10:33:56.305309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305757+0000","last_change":"2024-03-24T10:33:55.144173+0000","last_active":"2024-03-24T10:33:56.305757+0000","last_peered":"2024-03-24T10:33:56.305757+0000","last_clean":"2024-03-24T10:33:56.305757+0000","last_became_active":"2024-03-24T10:33:55.143874+0000","last_became_peered":"2024-03-24T10:33:55.143874+0000","last_unstale":"2024-03-24T10:33:56.305757+0000","last_undegraded":"2024-03-24T10:33:56.305757+0000","last_fullsized":"2024-03-24T10:33:56.305757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.136315+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.135994+0000","last_became_peered":"2024-03-24T10:33:55.135994+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124458+0000","last_change":"2024-03-24T10:33:55.159140+0000","last_active":"2024-03-24T10:33:56.124458+0000","last_peered":"2024-03-24T10:33:56.124458+0000","last_clean":"2024-03-24T10:33:56.124458+0000","last_became_active":"2024-03-24T10:33:55.158875+0000","last_became_peered":"2024-03-24T10:33:55.158875+0000","last_unstale":"2024-03-24T10:33:56.124458+0000","last_undegraded":"2024-03-24T10:33:56.124458+0000","last_fullsized":"2024-03-24T10:33:56.124458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123157+0000","last_change":"2024-03-24T10:33:55.134527+0000","last_active":"2024-03-24T10:33:56.123157+0000","last_peered":"2024-03-24T10:33:56.123157+0000","last_clean":"2024-03-24T10:33:56.123157+0000","last_became_active":"2024-03-24T10:33:55.134220+0000","last_became_peered":"2024-03-24T10:33:55.134220+0000","last_unstale":"2024-03-24T10:33:56.123157+0000","last_undegraded":"2024-03-24T10:33:56.123157+0000","last_fullsized":"2024-03-24T10:33:56.123157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306047+0000","last_change":"2024-03-24T10:33:55.138237+0000","last_active":"2024-03-24T10:33:56.306047+0000","last_peered":"2024-03-24T10:33:56.306047+0000","last_clean":"2024-03-24T10:33:56.306047+0000","last_became_active":"2024-03-24T10:33:55.136755+0000","last_became_peered":"2024-03-24T10:33:55.136755+0000","last_unstale":"2024-03-24T10:33:56.306047+0000","last_undegraded":"2024-03-24T10:33:56.306047+0000","last_fullsized":"2024-03-24T10:33:56.306047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305840+0000","last_change":"2024-03-24T10:33:55.145372+0000","last_active":"2024-03-24T10:33:56.305840+0000","last_peered":"2024-03-24T10:33:56.305840+0000","last_clean":"2024-03-24T10:33:56.305840+0000","last_became_active":"2024-03-24T10:33:55.144207+0000","last_became_peered":"2024-03-24T10:33:55.144207+0000","last_unstale":"2024-03-24T10:33:56.305840+0000","last_undegraded":"2024-03-24T10:33:56.305840+0000","last_fullsized":"2024-03-24T10:33:56.305840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306010+0000","last_change":"2024-03-24T10:33:55.146001+0000","last_active":"2024-03-24T10:33:56.306010+0000","last_peered":"2024-03-24T10:33:56.306010+0000","last_clean":"2024-03-24T10:33:56.306010+0000","last_became_active":"2024-03-24T10:33:55.145472+0000","last_became_peered":"2024-03-24T10:33:55.145472+0000","last_unstale":"2024-03-24T10:33:56.306010+0000","last_undegraded":"2024-03-24T10:33:56.306010+0000","last_fullsized":"2024-03-24T10:33:56.306010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123635+0000","last_change":"2024-03-24T10:33:55.143320+0000","last_active":"2024-03-24T10:33:56.123635+0000","last_peered":"2024-03-24T10:33:56.123635+0000","last_clean":"2024-03-24T10:33:56.123635+0000","last_became_active":"2024-03-24T10:33:55.143046+0000","last_became_peered":"2024-03-24T10:33:55.143046+0000","last_unstale":"2024-03-24T10:33:56.123635+0000","last_undegraded":"2024-03-24T10:33:56.123635+0000","last_fullsized":"2024-03-24T10:33:56.123635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124493+0000","last_change":"2024-03-24T10:33:55.147275+0000","last_active":"2024-03-24T10:33:56.124493+0000","last_peered":"2024-03-24T10:33:56.124493+0000","last_clean":"2024-03-24T10:33:56.124493+0000","last_became_active":"2024-03-24T10:33:55.146476+0000","last_became_peered":"2024-03-24T10:33:55.146476+0000","last_unstale":"2024-03-24T10:33:56.124493+0000","last_undegraded":"2024-03-24T10:33:56.124493+0000","last_fullsized":"2024-03-24T10:33:56.124493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305716+0000","last_change":"2024-03-24T10:33:55.131787+0000","last_active":"2024-03-24T10:33:56.305716+0000","last_peered":"2024-03-24T10:33:56.305716+0000","last_clean":"2024-03-24T10:33:56.305716+0000","last_became_active":"2024-03-24T10:33:55.131462+0000","last_became_peered":"2024-03-24T10:33:55.131462+0000","last_unstale":"2024-03-24T10:33:56.305716+0000","last_undegraded":"2024-03-24T10:33:56.305716+0000","last_fullsized":"2024-03-24T10:33:56.305716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:24.900927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305842+0000","last_change":"2024-03-24T10:33:54.130171+0000","last_active":"2024-03-24T10:33:56.305842+0000","last_peered":"2024-03-24T10:33:56.305842+0000","last_clean":"2024-03-24T10:33:56.305842+0000","last_became_active":"2024-03-24T10:33:54.129845+0000","last_became_peered":"2024-03-24T10:33:54.129845+0000","last_unstale":"2024-03-24T10:33:56.305842+0000","last_undegraded":"2024-03-24T10:33:56.305842+0000","last_fullsized":"2024-03-24T10:33:56.305842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305988+0000","last_change":"2024-03-24T10:33:55.156576+0000","last_active":"2024-03-24T10:33:56.305988+0000","last_peered":"2024-03-24T10:33:56.305988+0000","last_clean":"2024-03-24T10:33:56.305988+0000","last_became_active":"2024-03-24T10:33:55.156301+0000","last_became_peered":"2024-03-24T10:33:55.156301+0000","last_unstale":"2024-03-24T10:33:56.305988+0000","last_undegraded":"2024-03-24T10:33:56.305988+0000","last_fullsized":"2024-03-24T10:33:56.305988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124113+0000","last_change":"2024-03-24T10:33:54.122838+0000","last_active":"2024-03-24T10:33:56.124113+0000","last_peered":"2024-03-24T10:33:56.124113+0000","last_clean":"2024-03-24T10:33:56.124113+0000","last_became_active":"2024-03-24T10:33:54.122012+0000","last_became_peered":"2024-03-24T10:33:54.122012+0000","last_unstale":"2024-03-24T10:33:56.124113+0000","last_undegraded":"2024-03-24T10:33:56.124113+0000","last_fullsized":"2024-03-24T10:33:56.124113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305341+0000","last_change":"2024-03-24T10:33:54.127097+0000","last_active":"2024-03-24T10:33:56.305341+0000","last_peered":"2024-03-24T10:33:56.305341+0000","last_clean":"2024-03-24T10:33:56.305341+0000","last_became_active":"2024-03-24T10:33:54.126575+0000","last_became_peered":"2024-03-24T10:33:54.126575+0000","last_unstale":"2024-03-24T10:33:56.305341+0000","last_undegraded":"2024-03-24T10:33:56.305341+0000","last_fullsized":"2024-03-24T10:33:56.305341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305318+0000","last_change":"2024-03-24T10:33:55.138621+0000","last_active":"2024-03-24T10:33:56.305318+0000","last_peered":"2024-03-24T10:33:56.305318+0000","last_clean":"2024-03-24T10:33:56.305318+0000","last_became_active":"2024-03-24T10:33:55.138352+0000","last_became_peered":"2024-03-24T10:33:55.138352+0000","last_unstale":"2024-03-24T10:33:56.305318+0000","last_undegraded":"2024-03-24T10:33:56.305318+0000","last_fullsized":"2024-03-24T10:33:56.305318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305437+0000","last_change":"2024-03-24T10:33:54.125263+0000","last_active":"2024-03-24T10:33:56.305437+0000","last_peered":"2024-03-24T10:33:56.305437+0000","last_clean":"2024-03-24T10:33:56.305437+0000","last_became_active":"2024-03-24T10:33:54.124945+0000","last_became_peered":"2024-03-24T10:33:54.124945+0000","last_unstale":"2024-03-24T10:33:56.305437+0000","last_undegraded":"2024-03-24T10:33:56.305437+0000","last_fullsized":"2024-03-24T10:33:56.305437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122873+0000","last_change":"2024-03-24T10:33:55.134395+0000","last_active":"2024-03-24T10:33:56.122873+0000","last_peered":"2024-03-24T10:33:56.122873+0000","last_clean":"2024-03-24T10:33:56.122873+0000","last_became_active":"2024-03-24T10:33:55.133957+0000","last_became_peered":"2024-03-24T10:33:55.133957+0000","last_unstale":"2024-03-24T10:33:56.122873+0000","last_undegraded":"2024-03-24T10:33:56.122873+0000","last_fullsized":"2024-03-24T10:33:56.122873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124381+0000","last_change":"2024-03-24T10:33:54.127218+0000","last_active":"2024-03-24T10:33:56.124381+0000","last_peered":"2024-03-24T10:33:56.124381+0000","last_clean":"2024-03-24T10:33:56.124381+0000","last_became_active":"2024-03-24T10:33:54.126932+0000","last_became_peered":"2024-03-24T10:33:54.126932+0000","last_unstale":"2024-03-24T10:33:56.124381+0000","last_undegraded":"2024-03-24T10:33:56.124381+0000","last_fullsized":"2024-03-24T10:33:56.124381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306066+0000","last_change":"2024-03-24T10:33:55.150999+0000","last_active":"2024-03-24T10:33:56.306066+0000","last_peered":"2024-03-24T10:33:56.306066+0000","last_clean":"2024-03-24T10:33:56.306066+0000","last_became_active":"2024-03-24T10:33:55.150729+0000","last_became_peered":"2024-03-24T10:33:55.150729+0000","last_unstale":"2024-03-24T10:33:56.306066+0000","last_undegraded":"2024-03-24T10:33:56.306066+0000","last_fullsized":"2024-03-24T10:33:56.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306163+0000","last_change":"2024-03-24T10:33:54.126346+0000","last_active":"2024-03-24T10:33:56.306163+0000","last_peered":"2024-03-24T10:33:56.306163+0000","last_clean":"2024-03-24T10:33:56.306163+0000","last_became_active":"2024-03-24T10:33:54.126026+0000","last_became_peered":"2024-03-24T10:33:54.126026+0000","last_unstale":"2024-03-24T10:33:56.306163+0000","last_undegraded":"2024-03-24T10:33:56.306163+0000","last_fullsized":"2024-03-24T10:33:56.306163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306298+0000","last_change":"2024-03-24T10:33:55.145385+0000","last_active":"2024-03-24T10:33:56.306298+0000","last_peered":"2024-03-24T10:33:56.306298+0000","last_clean":"2024-03-24T10:33:56.306298+0000","last_became_active":"2024-03-24T10:33:55.144766+0000","last_became_peered":"2024-03-24T10:33:55.144766+0000","last_unstale":"2024-03-24T10:33:56.306298+0000","last_undegraded":"2024-03-24T10:33:56.306298+0000","last_fullsized":"2024-03-24T10:33:56.306298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123213+0000","last_change":"2024-03-24T10:33:54.128955+0000","last_active":"2024-03-24T10:33:56.123213+0000","last_peered":"2024-03-24T10:33:56.123213+0000","last_clean":"2024-03-24T10:33:56.123213+0000","last_became_active":"2024-03-24T10:33:54.128654+0000","last_became_peered":"2024-03-24T10:33:54.128654+0000","last_unstale":"2024-03-24T10:33:56.123213+0000","last_undegraded":"2024-03-24T10:33:56.123213+0000","last_fullsized":"2024-03-24T10:33:56.123213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306400+0000","last_change":"2024-03-24T10:33:55.146595+0000","last_active":"2024-03-24T10:33:56.306400+0000","last_peered":"2024-03-24T10:33:56.306400+0000","last_clean":"2024-03-24T10:33:56.306400+0000","last_became_active":"2024-03-24T10:33:55.146309+0000","last_became_peered":"2024-03-24T10:33:55.146309+0000","last_unstale":"2024-03-24T10:33:56.306400+0000","last_undegraded":"2024-03-24T10:33:56.306400+0000","last_fullsized":"2024-03-24T10:33:56.306400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124069+0000","last_change":"2024-03-24T10:33:54.125637+0000","last_active":"2024-03-24T10:33:56.124069+0000","last_peered":"2024-03-24T10:33:56.124069+0000","last_clean":"2024-03-24T10:33:56.124069+0000","last_became_active":"2024-03-24T10:33:54.125278+0000","last_became_peered":"2024-03-24T10:33:54.125278+0000","last_unstale":"2024-03-24T10:33:56.124069+0000","last_undegraded":"2024-03-24T10:33:56.124069+0000","last_fullsized":"2024-03-24T10:33:56.124069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306668+0000","last_change":"2024-03-24T10:33:55.156480+0000","last_active":"2024-03-24T10:33:56.306668+0000","last_peered":"2024-03-24T10:33:56.306668+0000","last_clean":"2024-03-24T10:33:56.306668+0000","last_became_active":"2024-03-24T10:33:55.156201+0000","last_became_peered":"2024-03-24T10:33:55.156201+0000","last_unstale":"2024-03-24T10:33:56.306668+0000","last_undegraded":"2024-03-24T10:33:56.306668+0000","last_fullsized":"2024-03-24T10:33:56.306668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306091+0000","last_change":"2024-03-24T10:33:54.125505+0000","last_active":"2024-03-24T10:33:56.306091+0000","last_peered":"2024-03-24T10:33:56.306091+0000","last_clean":"2024-03-24T10:33:56.306091+0000","last_became_active":"2024-03-24T10:33:54.125170+0000","last_became_peered":"2024-03-24T10:33:54.125170+0000","last_unstale":"2024-03-24T10:33:56.306091+0000","last_undegraded":"2024-03-24T10:33:56.306091+0000","last_fullsized":"2024-03-24T10:33:56.306091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305748+0000","last_change":"2024-03-24T10:33:55.156955+0000","last_active":"2024-03-24T10:33:56.305748+0000","last_peered":"2024-03-24T10:33:56.305748+0000","last_clean":"2024-03-24T10:33:56.305748+0000","last_became_active":"2024-03-24T10:33:55.156565+0000","last_became_peered":"2024-03-24T10:33:55.156565+0000","last_unstale":"2024-03-24T10:33:56.305748+0000","last_undegraded":"2024-03-24T10:33:56.305748+0000","last_fullsized":"2024-03-24T10:33:56.305748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:32.795374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306167+0000","last_change":"2024-03-24T10:33:54.128904+0000","last_active":"2024-03-24T10:33:56.306167+0000","last_peered":"2024-03-24T10:33:56.306167+0000","last_clean":"2024-03-24T10:33:56.306167+0000","last_became_active":"2024-03-24T10:33:54.128349+0000","last_became_peered":"2024-03-24T10:33:54.128349+0000","last_unstale":"2024-03-24T10:33:56.306167+0000","last_undegraded":"2024-03-24T10:33:56.306167+0000","last_fullsized":"2024-03-24T10:33:56.306167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306695+0000","last_change":"2024-03-24T10:33:55.155945+0000","last_active":"2024-03-24T10:33:56.306695+0000","last_peered":"2024-03-24T10:33:56.306695+0000","last_clean":"2024-03-24T10:33:56.306695+0000","last_became_active":"2024-03-24T10:33:55.155660+0000","last_became_peered":"2024-03-24T10:33:55.155660+0000","last_unstale":"2024-03-24T10:33:56.306695+0000","last_undegraded":"2024-03-24T10:33:56.306695+0000","last_fullsized":"2024-03-24T10:33:56.306695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124002+0000","last_change":"2024-03-24T10:33:54.127975+0000","last_active":"2024-03-24T10:33:56.124002+0000","last_peered":"2024-03-24T10:33:56.124002+0000","last_clean":"2024-03-24T10:33:56.124002+0000","last_became_active":"2024-03-24T10:33:54.127705+0000","last_became_peered":"2024-03-24T10:33:54.127705+0000","last_unstale":"2024-03-24T10:33:56.124002+0000","last_undegraded":"2024-03-24T10:33:56.124002+0000","last_fullsized":"2024-03-24T10:33:56.124002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124453+0000","last_change":"2024-03-24T10:33:55.146773+0000","last_active":"2024-03-24T10:33:56.124453+0000","last_peered":"2024-03-24T10:33:56.124453+0000","last_clean":"2024-03-24T10:33:56.124453+0000","last_became_active":"2024-03-24T10:33:55.146086+0000","last_became_peered":"2024-03-24T10:33:55.146086+0000","last_unstale":"2024-03-24T10:33:56.124453+0000","last_undegraded":"2024-03-24T10:33:56.124453+0000","last_fullsized":"2024-03-24T10:33:56.124453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123547+0000","last_change":"2024-03-24T10:33:54.114004+0000","last_active":"2024-03-24T10:33:56.123547+0000","last_peered":"2024-03-24T10:33:56.123547+0000","last_clean":"2024-03-24T10:33:56.123547+0000","last_became_active":"2024-03-24T10:33:54.113714+0000","last_became_peered":"2024-03-24T10:33:54.113714+0000","last_unstale":"2024-03-24T10:33:56.123547+0000","last_undegraded":"2024-03-24T10:33:56.123547+0000","last_fullsized":"2024-03-24T10:33:56.123547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306338+0000","last_change":"2024-03-24T10:33:55.160713+0000","last_active":"2024-03-24T10:33:56.306338+0000","last_peered":"2024-03-24T10:33:56.306338+0000","last_clean":"2024-03-24T10:33:56.306338+0000","last_became_active":"2024-03-24T10:33:55.157835+0000","last_became_peered":"2024-03-24T10:33:55.157835+0000","last_unstale":"2024-03-24T10:33:56.306338+0000","last_undegraded":"2024-03-24T10:33:56.306338+0000","last_fullsized":"2024-03-24T10:33:56.306338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306046+0000","last_change":"2024-03-24T10:33:54.123641+0000","last_active":"2024-03-24T10:33:56.306046+0000","last_peered":"2024-03-24T10:33:56.306046+0000","last_clean":"2024-03-24T10:33:56.306046+0000","last_became_active":"2024-03-24T10:33:54.123045+0000","last_became_peered":"2024-03-24T10:33:54.123045+0000","last_unstale":"2024-03-24T10:33:56.306046+0000","last_undegraded":"2024-03-24T10:33:56.306046+0000","last_fullsized":"2024-03-24T10:33:56.306046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:41:41.902140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306617+0000","last_change":"2024-03-24T10:33:55.155307+0000","last_active":"2024-03-24T10:33:56.306617+0000","last_peered":"2024-03-24T10:33:56.306617+0000","last_clean":"2024-03-24T10:33:56.306617+0000","last_became_active":"2024-03-24T10:33:55.154976+0000","last_became_peered":"2024-03-24T10:33:55.154976+0000","last_unstale":"2024-03-24T10:33:56.306617+0000","last_undegraded":"2024-03-24T10:33:56.306617+0000","last_fullsized":"2024-03-24T10:33:56.306617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123767+0000","last_change":"2024-03-24T10:33:54.124359+0000","last_active":"2024-03-24T10:33:56.123767+0000","last_peered":"2024-03-24T10:33:56.123767+0000","last_clean":"2024-03-24T10:33:56.123767+0000","last_became_active":"2024-03-24T10:33:54.124063+0000","last_became_peered":"2024-03-24T10:33:54.124063+0000","last_unstale":"2024-03-24T10:33:56.123767+0000","last_undegraded":"2024-03-24T10:33:56.123767+0000","last_fullsized":"2024-03-24T10:33:56.123767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306899+0000","last_change":"2024-03-24T10:33:55.159546+0000","last_active":"2024-03-24T10:33:56.306899+0000","last_peered":"2024-03-24T10:33:56.306899+0000","last_clean":"2024-03-24T10:33:56.306899+0000","last_became_active":"2024-03-24T10:33:55.159193+0000","last_became_peered":"2024-03-24T10:33:55.159193+0000","last_unstale":"2024-03-24T10:33:56.306899+0000","last_undegraded":"2024-03-24T10:33:56.306899+0000","last_fullsized":"2024-03-24T10:33:56.306899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:07.558259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123249+0000","last_change":"2024-03-24T10:33:54.127205+0000","last_active":"2024-03-24T10:33:56.123249+0000","last_peered":"2024-03-24T10:33:56.123249+0000","last_clean":"2024-03-24T10:33:56.123249+0000","last_became_active":"2024-03-24T10:33:54.126938+0000","last_became_peered":"2024-03-24T10:33:54.126938+0000","last_unstale":"2024-03-24T10:33:56.123249+0000","last_undegraded":"2024-03-24T10:33:56.123249+0000","last_fullsized":"2024-03-24T10:33:56.123249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305970+0000","last_change":"2024-03-24T10:33:55.150130+0000","last_active":"2024-03-24T10:33:56.305970+0000","last_peered":"2024-03-24T10:33:56.305970+0000","last_clean":"2024-03-24T10:33:56.305970+0000","last_became_active":"2024-03-24T10:33:55.145689+0000","last_became_peered":"2024-03-24T10:33:55.145689+0000","last_unstale":"2024-03-24T10:33:56.305970+0000","last_undegraded":"2024-03-24T10:33:56.305970+0000","last_fullsized":"2024-03-24T10:33:56.305970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123759+0000","last_change":"2024-03-24T10:33:54.131439+0000","last_active":"2024-03-24T10:33:56.123759+0000","last_peered":"2024-03-24T10:33:56.123759+0000","last_clean":"2024-03-24T10:33:56.123759+0000","last_became_active":"2024-03-24T10:33:54.131117+0000","last_became_peered":"2024-03-24T10:33:54.131117+0000","last_unstale":"2024-03-24T10:33:56.123759+0000","last_undegraded":"2024-03-24T10:33:56.123759+0000","last_fullsized":"2024-03-24T10:33:56.123759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306791+0000","last_change":"2024-03-24T10:33:55.157088+0000","last_active":"2024-03-24T10:33:56.306791+0000","last_peered":"2024-03-24T10:33:56.306791+0000","last_clean":"2024-03-24T10:33:56.306791+0000","last_became_active":"2024-03-24T10:33:55.156790+0000","last_became_peered":"2024-03-24T10:33:55.156790+0000","last_unstale":"2024-03-24T10:33:56.306791+0000","last_undegraded":"2024-03-24T10:33:56.306791+0000","last_fullsized":"2024-03-24T10:33:56.306791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306529+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:33:56.306529+0000","last_peered":"2024-03-24T10:33:56.306529+0000","last_clean":"2024-03-24T10:33:56.306529+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:33:56.306529+0000","last_undegraded":"2024-03-24T10:33:56.306529+0000","last_fullsized":"2024-03-24T10:33:56.306529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305276+0000","last_change":"2024-03-24T10:33:54.121820+0000","last_active":"2024-03-24T10:33:56.305276+0000","last_peered":"2024-03-24T10:33:56.305276+0000","last_clean":"2024-03-24T10:33:56.305276+0000","last_became_active":"2024-03-24T10:33:54.121496+0000","last_became_peered":"2024-03-24T10:33:54.121496+0000","last_unstale":"2024-03-24T10:33:56.305276+0000","last_undegraded":"2024-03-24T10:33:56.305276+0000","last_fullsized":"2024-03-24T10:33:56.305276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305400+0000","last_change":"2024-03-24T10:33:55.146810+0000","last_active":"2024-03-24T10:33:56.305400+0000","last_peered":"2024-03-24T10:33:56.305400+0000","last_clean":"2024-03-24T10:33:56.305400+0000","last_became_active":"2024-03-24T10:33:55.146508+0000","last_became_peered":"2024-03-24T10:33:55.146508+0000","last_unstale":"2024-03-24T10:33:56.305400+0000","last_undegraded":"2024-03-24T10:33:56.305400+0000","last_fullsized":"2024-03-24T10:33:56.305400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306540+0000","last_change":"2024-03-24T10:33:54.127727+0000","last_active":"2024-03-24T10:33:56.306540+0000","last_peered":"2024-03-24T10:33:56.306540+0000","last_clean":"2024-03-24T10:33:56.306540+0000","last_became_active":"2024-03-24T10:33:54.127106+0000","last_became_peered":"2024-03-24T10:33:54.127106+0000","last_unstale":"2024-03-24T10:33:56.306540+0000","last_undegraded":"2024-03-24T10:33:56.306540+0000","last_fullsized":"2024-03-24T10:33:56.306540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:31.791505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306196+0000","last_change":"2024-03-24T10:33:55.156425+0000","last_active":"2024-03-24T10:33:56.306196+0000","last_peered":"2024-03-24T10:33:56.306196+0000","last_clean":"2024-03-24T10:33:56.306196+0000","last_became_active":"2024-03-24T10:33:55.156146+0000","last_became_peered":"2024-03-24T10:33:55.156146+0000","last_unstale":"2024-03-24T10:33:56.306196+0000","last_undegraded":"2024-03-24T10:33:56.306196+0000","last_fullsized":"2024-03-24T10:33:56.306196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123643+0000","last_change":"2024-03-24T10:33:54.127887+0000","last_active":"2024-03-24T10:33:56.123643+0000","last_peered":"2024-03-24T10:33:56.123643+0000","last_clean":"2024-03-24T10:33:56.123643+0000","last_became_active":"2024-03-24T10:33:54.127602+0000","last_became_peered":"2024-03-24T10:33:54.127602+0000","last_unstale":"2024-03-24T10:33:56.123643+0000","last_undegraded":"2024-03-24T10:33:56.123643+0000","last_fullsized":"2024-03-24T10:33:56.123643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122824+0000","last_change":"2024-03-24T10:33:55.128811+0000","last_active":"2024-03-24T10:33:56.122824+0000","last_peered":"2024-03-24T10:33:56.122824+0000","last_clean":"2024-03-24T10:33:56.122824+0000","last_became_active":"2024-03-24T10:33:55.128094+0000","last_became_peered":"2024-03-24T10:33:55.128094+0000","last_unstale":"2024-03-24T10:33:56.122824+0000","last_undegraded":"2024-03-24T10:33:56.122824+0000","last_fullsized":"2024-03-24T10:33:56.122824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306104+0000","last_change":"2024-03-24T10:33:54.123812+0000","last_active":"2024-03-24T10:33:56.306104+0000","last_peered":"2024-03-24T10:33:56.306104+0000","last_clean":"2024-03-24T10:33:56.306104+0000","last_became_active":"2024-03-24T10:33:54.122614+0000","last_became_peered":"2024-03-24T10:33:54.122614+0000","last_unstale":"2024-03-24T10:33:56.306104+0000","last_undegraded":"2024-03-24T10:33:56.306104+0000","last_fullsized":"2024-03-24T10:33:56.306104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305946+0000","last_change":"2024-03-24T10:33:55.140461+0000","last_active":"2024-03-24T10:33:56.305946+0000","last_peered":"2024-03-24T10:33:56.305946+0000","last_clean":"2024-03-24T10:33:56.305946+0000","last_became_active":"2024-03-24T10:33:55.139737+0000","last_became_peered":"2024-03-24T10:33:55.139737+0000","last_unstale":"2024-03-24T10:33:56.305946+0000","last_undegraded":"2024-03-24T10:33:56.305946+0000","last_fullsized":"2024-03-24T10:33:56.305946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:27.228493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122944+0000","last_change":"2024-03-24T10:33:54.123506+0000","last_active":"2024-03-24T10:33:56.122944+0000","last_peered":"2024-03-24T10:33:56.122944+0000","last_clean":"2024-03-24T10:33:56.122944+0000","last_became_active":"2024-03-24T10:33:54.123194+0000","last_became_peered":"2024-03-24T10:33:54.123194+0000","last_unstale":"2024-03-24T10:33:56.122944+0000","last_undegraded":"2024-03-24T10:33:56.122944+0000","last_fullsized":"2024-03-24T10:33:56.122944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305900+0000","last_change":"2024-03-24T10:33:55.148696+0000","last_active":"2024-03-24T10:33:56.305900+0000","last_peered":"2024-03-24T10:33:56.305900+0000","last_clean":"2024-03-24T10:33:56.305900+0000","last_became_active":"2024-03-24T10:33:55.148382+0000","last_became_peered":"2024-03-24T10:33:55.148382+0000","last_unstale":"2024-03-24T10:33:56.305900+0000","last_undegraded":"2024-03-24T10:33:56.305900+0000","last_fullsized":"2024-03-24T10:33:56.305900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306970+0000","last_change":"2024-03-24T10:33:54.129296+0000","last_active":"2024-03-24T10:33:56.306970+0000","last_peered":"2024-03-24T10:33:56.306970+0000","last_clean":"2024-03-24T10:33:56.306970+0000","last_became_active":"2024-03-24T10:33:54.128788+0000","last_became_peered":"2024-03-24T10:33:54.128788+0000","last_unstale":"2024-03-24T10:33:56.306970+0000","last_undegraded":"2024-03-24T10:33:56.306970+0000","last_fullsized":"2024-03-24T10:33:56.306970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:48.412131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122832+0000","last_change":"2024-03-24T10:33:55.128461+0000","last_active":"2024-03-24T10:33:56.122832+0000","last_peered":"2024-03-24T10:33:56.122832+0000","last_clean":"2024-03-24T10:33:56.122832+0000","last_became_active":"2024-03-24T10:33:55.128125+0000","last_became_peered":"2024-03-24T10:33:55.128125+0000","last_unstale":"2024-03-24T10:33:56.122832+0000","last_undegraded":"2024-03-24T10:33:56.122832+0000","last_fullsized":"2024-03-24T10:33:56.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.140759+0000","last_change":"2024-03-24T10:33:54.117337+0000","last_active":"2024-03-24T10:33:56.140759+0000","last_peered":"2024-03-24T10:33:56.140759+0000","last_clean":"2024-03-24T10:33:56.140759+0000","last_became_active":"2024-03-24T10:33:54.116814+0000","last_became_peered":"2024-03-24T10:33:54.116814+0000","last_unstale":"2024-03-24T10:33:56.140759+0000","last_undegraded":"2024-03-24T10:33:56.140759+0000","last_fullsized":"2024-03-24T10:33:56.140759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.147551+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.147261+0000","last_became_peered":"2024-03-24T10:33:55.147261+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124432+0000","last_change":"2024-03-24T10:33:54.127434+0000","last_active":"2024-03-24T10:33:56.124432+0000","last_peered":"2024-03-24T10:33:56.124432+0000","last_clean":"2024-03-24T10:33:56.124432+0000","last_became_active":"2024-03-24T10:33:54.127162+0000","last_became_peered":"2024-03-24T10:33:54.127162+0000","last_unstale":"2024-03-24T10:33:56.124432+0000","last_undegraded":"2024-03-24T10:33:56.124432+0000","last_fullsized":"2024-03-24T10:33:56.124432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306039+0000","last_change":"2024-03-24T10:33:55.142227+0000","last_active":"2024-03-24T10:33:56.306039+0000","last_peered":"2024-03-24T10:33:56.306039+0000","last_clean":"2024-03-24T10:33:56.306039+0000","last_became_active":"2024-03-24T10:33:55.141920+0000","last_became_peered":"2024-03-24T10:33:55.141920+0000","last_unstale":"2024-03-24T10:33:56.306039+0000","last_undegraded":"2024-03-24T10:33:56.306039+0000","last_fullsized":"2024-03-24T10:33:56.306039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306452+0000","last_change":"2024-03-24T10:33:54.128286+0000","last_active":"2024-03-24T10:33:56.306452+0000","last_peered":"2024-03-24T10:33:56.306452+0000","last_clean":"2024-03-24T10:33:56.306452+0000","last_became_active":"2024-03-24T10:33:54.127434+0000","last_became_peered":"2024-03-24T10:33:54.127434+0000","last_unstale":"2024-03-24T10:33:56.306452+0000","last_undegraded":"2024-03-24T10:33:56.306452+0000","last_fullsized":"2024-03-24T10:33:56.306452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:40.602694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305662+0000","last_change":"2024-03-24T10:33:55.132582+0000","last_active":"2024-03-24T10:33:56.305662+0000","last_peered":"2024-03-24T10:33:56.305662+0000","last_clean":"2024-03-24T10:33:56.305662+0000","last_became_active":"2024-03-24T10:33:55.132259+0000","last_became_peered":"2024-03-24T10:33:55.132259+0000","last_unstale":"2024-03-24T10:33:56.305662+0000","last_undegraded":"2024-03-24T10:33:56.305662+0000","last_fullsized":"2024-03-24T10:33:56.305662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:22.596121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122894+0000","last_change":"2024-03-24T10:33:54.127899+0000","last_active":"2024-03-24T10:33:56.122894+0000","last_peered":"2024-03-24T10:33:56.122894+0000","last_clean":"2024-03-24T10:33:56.122894+0000","last_became_active":"2024-03-24T10:33:54.127592+0000","last_became_peered":"2024-03-24T10:33:54.127592+0000","last_unstale":"2024-03-24T10:33:56.122894+0000","last_undegraded":"2024-03-24T10:33:56.122894+0000","last_fullsized":"2024-03-24T10:33:56.122894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305592+0000","last_change":"2024-03-24T10:33:55.133853+0000","last_active":"2024-03-24T10:33:56.305592+0000","last_peered":"2024-03-24T10:33:56.305592+0000","last_clean":"2024-03-24T10:33:56.305592+0000","last_became_active":"2024-03-24T10:33:55.133478+0000","last_became_peered":"2024-03-24T10:33:55.133478+0000","last_unstale":"2024-03-24T10:33:56.305592+0000","last_undegraded":"2024-03-24T10:33:56.305592+0000","last_fullsized":"2024-03-24T10:33:56.305592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123001+0000","last_change":"2024-03-24T10:33:54.126665+0000","last_active":"2024-03-24T10:33:56.123001+0000","last_peered":"2024-03-24T10:33:56.123001+0000","last_clean":"2024-03-24T10:33:56.123001+0000","last_became_active":"2024-03-24T10:33:54.126339+0000","last_became_peered":"2024-03-24T10:33:54.126339+0000","last_unstale":"2024-03-24T10:33:56.123001+0000","last_undegraded":"2024-03-24T10:33:56.123001+0000","last_fullsized":"2024-03-24T10:33:56.123001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305669+0000","last_change":"2024-03-24T10:33:55.140585+0000","last_active":"2024-03-24T10:33:56.305669+0000","last_peered":"2024-03-24T10:33:56.305669+0000","last_clean":"2024-03-24T10:33:56.305669+0000","last_became_active":"2024-03-24T10:33:55.140142+0000","last_became_peered":"2024-03-24T10:33:55.140142+0000","last_unstale":"2024-03-24T10:33:56.305669+0000","last_undegraded":"2024-03-24T10:33:56.305669+0000","last_fullsized":"2024-03-24T10:33:56.305669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306757+0000","last_change":"2024-03-24T10:33:54.128102+0000","last_active":"2024-03-24T10:33:56.306757+0000","last_peered":"2024-03-24T10:33:56.306757+0000","last_clean":"2024-03-24T10:33:56.306757+0000","last_became_active":"2024-03-24T10:33:54.127077+0000","last_became_peered":"2024-03-24T10:33:54.127077+0000","last_unstale":"2024-03-24T10:33:56.306757+0000","last_undegraded":"2024-03-24T10:33:56.306757+0000","last_fullsized":"2024-03-24T10:33:56.306757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123921+0000","last_change":"2024-03-24T10:33:55.155558+0000","last_active":"2024-03-24T10:33:56.123921+0000","last_peered":"2024-03-24T10:33:56.123921+0000","last_clean":"2024-03-24T10:33:56.123921+0000","last_became_active":"2024-03-24T10:33:55.155291+0000","last_became_peered":"2024-03-24T10:33:55.155291+0000","last_unstale":"2024-03-24T10:33:56.123921+0000","last_undegraded":"2024-03-24T10:33:56.123921+0000","last_fullsized":"2024-03-24T10:33:56.123921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305743+0000","last_change":"2024-03-24T10:33:55.159176+0000","last_active":"2024-03-24T10:33:56.305743+0000","last_peered":"2024-03-24T10:33:56.305743+0000","last_clean":"2024-03-24T10:33:56.305743+0000","last_became_active":"2024-03-24T10:33:55.158885+0000","last_became_peered":"2024-03-24T10:33:55.158885+0000","last_unstale":"2024-03-24T10:33:56.305743+0000","last_undegraded":"2024-03-24T10:33:56.305743+0000","last_fullsized":"2024-03-24T10:33:56.305743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306353+0000","last_change":"2024-03-24T10:33:54.132298+0000","last_active":"2024-03-24T10:33:56.306353+0000","last_peered":"2024-03-24T10:33:56.306353+0000","last_clean":"2024-03-24T10:33:56.306353+0000","last_became_active":"2024-03-24T10:33:54.132009+0000","last_became_peered":"2024-03-24T10:33:54.132009+0000","last_unstale":"2024-03-24T10:33:56.306353+0000","last_undegraded":"2024-03-24T10:33:56.306353+0000","last_fullsized":"2024-03-24T10:33:56.306353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305825+0000","last_change":"2024-03-24T10:33:55.159162+0000","last_active":"2024-03-24T10:33:56.305825+0000","last_peered":"2024-03-24T10:33:56.305825+0000","last_clean":"2024-03-24T10:33:56.305825+0000","last_became_active":"2024-03-24T10:33:55.158898+0000","last_became_peered":"2024-03-24T10:33:55.158898+0000","last_unstale":"2024-03-24T10:33:56.305825+0000","last_undegraded":"2024-03-24T10:33:56.305825+0000","last_fullsized":"2024-03-24T10:33:56.305825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306863+0000","last_change":"2024-03-24T10:33:54.129068+0000","last_active":"2024-03-24T10:33:56.306863+0000","last_peered":"2024-03-24T10:33:56.306863+0000","last_clean":"2024-03-24T10:33:56.306863+0000","last_became_active":"2024-03-24T10:33:54.128465+0000","last_became_peered":"2024-03-24T10:33:54.128465+0000","last_unstale":"2024-03-24T10:33:56.306863+0000","last_undegraded":"2024-03-24T10:33:56.306863+0000","last_fullsized":"2024-03-24T10:33:56.306863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306248+0000","last_change":"2024-03-24T10:33:55.146210+0000","last_active":"2024-03-24T10:33:56.306248+0000","last_peered":"2024-03-24T10:33:56.306248+0000","last_clean":"2024-03-24T10:33:56.306248+0000","last_became_active":"2024-03-24T10:33:55.145069+0000","last_became_peered":"2024-03-24T10:33:55.145069+0000","last_unstale":"2024-03-24T10:33:56.306248+0000","last_undegraded":"2024-03-24T10:33:56.306248+0000","last_fullsized":"2024-03-24T10:33:56.306248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305927+0000","last_change":"2024-03-24T10:33:54.126679+0000","last_active":"2024-03-24T10:33:56.305927+0000","last_peered":"2024-03-24T10:33:56.305927+0000","last_clean":"2024-03-24T10:33:56.305927+0000","last_became_active":"2024-03-24T10:33:54.126407+0000","last_became_peered":"2024-03-24T10:33:54.126407+0000","last_unstale":"2024-03-24T10:33:56.305927+0000","last_undegraded":"2024-03-24T10:33:56.305927+0000","last_fullsized":"2024-03-24T10:33:56.305927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305615+0000","last_change":"2024-03-24T10:33:55.150245+0000","last_active":"2024-03-24T10:33:56.305615+0000","last_peered":"2024-03-24T10:33:56.305615+0000","last_clean":"2024-03-24T10:33:56.305615+0000","last_became_active":"2024-03-24T10:33:55.149927+0000","last_became_peered":"2024-03-24T10:33:55.149927+0000","last_unstale":"2024-03-24T10:33:56.305615+0000","last_undegraded":"2024-03-24T10:33:56.305615+0000","last_fullsized":"2024-03-24T10:33:56.305615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306337+0000","last_change":"2024-03-24T10:33:54.127624+0000","last_active":"2024-03-24T10:33:56.306337+0000","last_peered":"2024-03-24T10:33:56.306337+0000","last_clean":"2024-03-24T10:33:56.306337+0000","last_became_active":"2024-03-24T10:33:54.127307+0000","last_became_peered":"2024-03-24T10:33:54.127307+0000","last_unstale":"2024-03-24T10:33:56.306337+0000","last_undegraded":"2024-03-24T10:33:56.306337+0000","last_fullsized":"2024-03-24T10:33:56.306337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":70917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":69465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:33:59.480 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T10:33:59.480 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T10:33:59.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-24T10:33:59.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-24T10:33:59.481 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-24T10:33:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 Processor -- start 2024-03-24T10:33:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 -- start start 2024-03-24T10:33:59.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4d40b7530 0x7fd4d40d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd4d40b6a40 0x7fd4d406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d40b4c70 con 0x7fd4d406cba0 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d40d1320 con 0x7fd4d40b7530 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.616+0000 7fd4da83b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d406d550 con 0x7fd4d40b6a40 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4d40b7530 0x7fd4d40d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4d40b7530 0x7fd4d40d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45404/0 (socket says 172.21.15.1:45404) 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 -- 172.21.15.1:0/582627343 learned_addr learned my addr 172.21.15.1:0/582627343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d8dd8640 1 --2- 172.21.15.1:0/582627343 >> v2:172.21.15.3:3300/0 conn(0x7fd4d40b6a40 0x7fd4d406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 -- 172.21.15.1:0/582627343 >> v2:172.21.15.3:3300/0 conn(0x7fd4d40b6a40 msgr2=0x7fd4d406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 --2- 172.21.15.1:0/582627343 >> v2:172.21.15.3:3300/0 conn(0x7fd4d40b6a40 0x7fd4d406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 -- 172.21.15.1:0/582627343 >> v1:172.21.15.3:6789/0 conn(0x7fd4d406cba0 legacy=0x7fd4d4161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d3fff640 1 -- 172.21.15.1:0/582627343 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d40b4f50 con 0x7fd4d40b7530 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.617+0000 7fd4d2ffd640 1 -- 172.21.15.1:0/582627343 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874896662 0 0) 0x7fd4d40b4c70 con 0x7fd4d406cba0 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4d3fff640 1 --2- 172.21.15.1:0/582627343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d40d09b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40cfc80 tx=0x7fd4c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999cf3e6b8301ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4d2ffd640 1 -- 172.21.15.1:0/582627343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4030860 con 0x7fd4d40b7530 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4d2ffd640 1 -- 172.21.15.1:0/582627343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c40309c0 con 0x7fd4d40b7530 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4d2ffd640 1 -- 172.21.15.1:0/582627343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4030d50 con 0x7fd4d40b7530 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 -- 172.21.15.1:0/582627343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 msgr2=0x7fd4d40d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 --2- 172.21.15.1:0/582627343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d40d09b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40cfc80 tx=0x7fd4c402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 -- 172.21.15.1:0/582627343 shutdown_connections 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 --2- 172.21.15.1:0/582627343 >> v2:172.21.15.3:3300/0 conn(0x7fd4d40b6a40 0x7fd4d406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 --2- 172.21.15.1:0/582627343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d40d09b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 -- 172.21.15.1:0/582627343 >> 172.21.15.1:0/582627343 conn(0x7fd4d40d1c80 msgr2=0x7fd4d40d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 -- 172.21.15.1:0/582627343 shutdown_connections 2024-03-24T10:33:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.618+0000 7fd4da83b640 1 -- 172.21.15.1:0/582627343 wait complete. 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 Processor -- start 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 -- start start 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4d406cba0 0x7fd4d4152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd4d40b6a40 0x7fd4d409e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d4153430 con 0x7fd4d40b7530 2024-03-24T10:33:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d409bd20 con 0x7fd4d406cba0 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.620+0000 7fd4da83b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d415f0a0 con 0x7fd4d40b6a40 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45406/0 (socket says 172.21.15.1:45406) 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4d406cba0 0x7fd4d4152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 -- 172.21.15.1:0/3636804807 learned_addr learned my addr 172.21.15.1:0/3636804807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d37fe640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd4d40b6a40 0x7fd4d409e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 -- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd4d40b6a40 msgr2=0x7fd4d409e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd4d40b6a40 0x7fd4d409e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 -- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4d406cba0 msgr2=0x7fd4d4152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4d406cba0 0x7fd4d4152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 -- 172.21.15.1:0/3636804807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c40094d0 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d8dd8640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00efc0 tx=0x7fd4cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc014480 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4cc00f040 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d409bfc0 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc014820 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.621+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d415f380 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.622+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d406de20 con 0x7fd4d40b7530 2024-03-24T10:33:59.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.623+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4cc014980 con 0x7fd4d40b7530 2024-03-24T10:33:59.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.623+0000 7fd4d0ff9640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 0x7fd4c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.623+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd4cc08af70 con 0x7fd4d40b7530 2024-03-24T10:33:59.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.623+0000 7fd4d3fff640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 0x7fd4c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.624+0000 7fd4d3fff640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 0x7fd4c006dcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4036000 tx=0x7fd4c4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.627+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4cc0bee10 con 0x7fd4d40b7530 2024-03-24T10:33:59.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.856+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd4d409c4e0 con 0x7fd4d40b7530 2024-03-24T10:33:59.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.859+0000 7fd4d0ff9640 1 -- 172.21.15.1:0/3636804807 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+171775 (secure 0 0 0) 0x7fd4cc0bbf10 con 0x7fd4d40b7530 2024-03-24T10:33:59.863 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:33:59.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 msgr2=0x7fd4c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 0x7fd4c006dcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4036000 tx=0x7fd4c4008030 comp rx=0 tx=0).stop 2024-03-24T10:33:59.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 msgr2=0x7fd4d409ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00efc0 tx=0x7fd4cc00c490 comp rx=0 tx=0).stop 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 shutdown_connections 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd4c006b810 0x7fd4c006dcd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4d40b7530 0x7fd4d409ead0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd4d40b6a40 0x7fd4d409e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 --2- 172.21.15.1:0/3636804807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4d406cba0 0x7fd4d4152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 >> 172.21.15.1:0/3636804807 conn(0x7fd4d40d1c80 msgr2=0x7fd4d40d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 shutdown_connections 2024-03-24T10:33:59.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.861+0000 7fd4da83b640 1 -- 172.21.15.1:0/3636804807 wait complete. 2024-03-24T10:33:59.873 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":2979692092},{"type":"v1","addr":"172.21.15.1:6833","nonce":2979692092}]},"active_addr":"172.21.15.1:6833/2979692092","active_change":"2024-03-24T10:33:34.508524+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":501110821}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3606450285}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2888614076}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":85794081}]}]} 2024-03-24T10:33:59.877 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-24T10:33:59.877 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-24T10:33:59.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.976+0000 7f1cc072c640 1 Processor -- start 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.976+0000 7f1cc072c640 1 -- start start 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cc072c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 0x7f1cb80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cc072c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cb806cba0 0x7f1cb806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cc072c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cb80b4c70 con 0x7f1cb8161c60 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cc072c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1cb80d1320 con 0x7f1cb80b7530 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cc072c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f1cb806d550 con 0x7f1cb806cba0 2024-03-24T10:33:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbe4c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 0x7f1cb80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbdcc7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cb806cba0 0x7f1cb806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbdcc7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cb806cba0 0x7f1cb806d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:50856/0 (socket says 172.21.15.1:50856) 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbecc9640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1cb8161c60 0x7f1cb8162060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:36152/0 (socket says 172.21.15.1:36152) 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbdcc7640 1 -- 172.21.15.1:0/2535742784 learned_addr learned my addr 172.21.15.1:0/2535742784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.977+0000 7f1cbe4c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 0x7f1cb80d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45408/0 (socket says 172.21.15.1:45408) 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbdcc7640 1 -- 172.21.15.1:0/2535742784 >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 msgr2=0x7f1cb80d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbdcc7640 1 --2- 172.21.15.1:0/2535742784 >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 0x7f1cb80d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbdcc7640 1 -- 172.21.15.1:0/2535742784 >> v1:172.21.15.3:6789/0 conn(0x7f1cb8161c60 legacy=0x7f1cb8162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbd4c6640 1 -- 172.21.15.1:0/2535742784 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825521262 0 0) 0x7f1cb80b4c70 con 0x7f1cb8161c60 2024-03-24T10:33:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbdcc7640 1 -- 172.21.15.1:0/2535742784 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb80b4f50 con 0x7f1cb806cba0 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.978+0000 7f1cbdcc7640 1 --2- 172.21.15.1:0/2535742784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb806cba0 0x7f1cb806d010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cb806c080 tx=0x7f1ca802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=679f5329fb9e3aad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cbd4c6640 1 -- 172.21.15.1:0/2535742784 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8030860 con 0x7f1cb806cba0 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cbd4c6640 1 -- 172.21.15.1:0/2535742784 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca80309c0 con 0x7f1cb806cba0 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cbd4c6640 1 -- 172.21.15.1:0/2535742784 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8030d50 con 0x7f1cb806cba0 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 -- 172.21.15.1:0/2535742784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb806cba0 msgr2=0x7f1cb806d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 --2- 172.21.15.1:0/2535742784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb806cba0 0x7f1cb806d010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cb806c080 tx=0x7f1ca802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 -- 172.21.15.1:0/2535742784 shutdown_connections 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 --2- 172.21.15.1:0/2535742784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb806cba0 0x7f1cb806d010 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 --2- 172.21.15.1:0/2535742784 >> v2:172.21.15.1:3300/0 conn(0x7f1cb80b7530 0x7f1cb80d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 -- 172.21.15.1:0/2535742784 >> 172.21.15.1:0/2535742784 conn(0x7f1cb80d1c80 msgr2=0x7f1cb80d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 -- 172.21.15.1:0/2535742784 shutdown_connections 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.979+0000 7f1cc072c640 1 -- 172.21.15.1:0/2535742784 wait complete. 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.980+0000 7f1cc072c640 1 Processor -- start 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.980+0000 7f1cc072c640 1 -- start start 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80b7530 0x7f1cb8153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1cb8161c60 0x7f1cb80c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb80b29d0 con 0x7f1cb806cba0 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb80b1930 con 0x7f1cb80b7530 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cc072c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb80b1c10 con 0x7f1cb8161c60 2024-03-24T10:33:59.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45414/0 (socket says 172.21.15.1:45414) 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 -- 172.21.15.1:0/4239069842 learned_addr learned my addr 172.21.15.1:0/4239069842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbdcc7640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80b7530 0x7f1cb8153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbecc9640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1cb8161c60 0x7f1cb80c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 -- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1cb8161c60 msgr2=0x7f1cb80c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1cb8161c60 0x7f1cb80c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 -- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80b7530 msgr2=0x7f1cb8153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80b7530 0x7f1cb8153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.981+0000 7f1cbe4c8640 1 -- 172.21.15.1:0/4239069842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca80094d0 con 0x7f1cb806cba0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1cbe4c8640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400e9b0 tx=0x7f1cb400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4009670 con 0x7f1cb806cba0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb400c9a0 con 0x7f1cb806cba0 2024-03-24T10:33:59.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb400cd50 con 0x7f1cb806cba0 2024-03-24T10:33:59.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb80b1ef0 con 0x7f1cb806cba0 2024-03-24T10:33:59.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.982+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb80b24b0 con 0x7f1cb806cba0 2024-03-24T10:33:59.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.983+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb806de20 con 0x7f1cb806cba0 2024-03-24T10:33:59.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.983+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1cb4013470 con 0x7f1cb806cba0 2024-03-24T10:33:59.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.984+0000 7f1caeffd640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 0x7f1c9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:59.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.984+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1cb408a440 con 0x7f1cb806cba0 2024-03-24T10:33:59.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.984+0000 7f1cbdcc7640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 0x7f1c9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:59.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.984+0000 7f1cbdcc7640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 0x7f1c9c06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cb806c080 tx=0x7f1ca8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:59.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:33:59.988+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1cb4054e90 con 0x7f1cb806cba0 2024-03-24T10:34:00.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.169+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cb804f4e0 con 0x7f1cb806cba0 2024-03-24T10:34:00.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.170+0000 7f1caeffd640 1 -- 172.21.15.1:0/4239069842 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f1cb4058550 con 0x7f1cb806cba0 2024-03-24T10:34:00.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:00.172 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 msgr2=0x7f1c9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 0x7f1c9c06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cb806c080 tx=0x7f1ca8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 msgr2=0x7f1cb80aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400e9b0 tx=0x7f1cb400ee70 comp rx=0 tx=0).stop 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 shutdown_connections 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1c9c06b810 0x7f1c9c06dcd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1cb8161c60 0x7f1cb80c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80b7530 0x7f1cb8153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 --2- 172.21.15.1:0/4239069842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cb806cba0 0x7f1cb80aea50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 >> 172.21.15.1:0/4239069842 conn(0x7f1cb80d1c80 msgr2=0x7f1cb80b6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 shutdown_connections 2024-03-24T10:34:00.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.172+0000 7f1cc072c640 1 -- 172.21.15.1:0/4239069842 wait complete. 2024-03-24T10:34:00.185 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-24T10:34:00.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:00.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.285+0000 7f516337a640 1 Processor -- start 2024-03-24T10:34:00.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.285+0000 7f516337a640 1 -- start start 2024-03-24T10:34:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.285+0000 7f516337a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f515c0b7530 0x7f515c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.285+0000 7f516337a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 0x7f515c06d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f516337a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515c0b4c70 con 0x7f515c06cba0 2024-03-24T10:34:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f516337a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f515c0d1320 con 0x7f515c0b7530 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f516337a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f515c06d550 con 0x7f515c0b6a40 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f515c0b7530 0x7f515c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5160915640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f515c06cba0 0x7f515c161c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39612/0 (socket says 172.21.15.1:39612) 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f515c0b7530 0x7f515c0d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45424/0 (socket says 172.21.15.1:45424) 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161917640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 0x7f515c06d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5160915640 1 -- 172.21.15.1:0/395341869 learned_addr learned my addr 172.21.15.1:0/395341869 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 -- 172.21.15.1:0/395341869 >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 msgr2=0x7f515c06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 --2- 172.21.15.1:0/395341869 >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 0x7f515c06d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 -- 172.21.15.1:0/395341869 >> v1:172.21.15.1:6789/0 conn(0x7f515c06cba0 legacy=0x7f515c161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161116640 1 -- 172.21.15.1:0/395341869 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c0b4f50 con 0x7f515c0b7530 2024-03-24T10:34:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.286+0000 7f5161917640 1 --2- 172.21.15.1:0/395341869 >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 0x7f515c06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:00.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f5161116640 1 --2- 172.21.15.1:0/395341869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b7530 0x7f515c0d09b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f515c0cfc80 tx=0x7f515802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73aaa8aec3a80e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f5153fff640 1 -- 172.21.15.1:0/395341869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158031830 con 0x7f515c0b7530 2024-03-24T10:34:00.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f5153fff640 1 -- 172.21.15.1:0/395341869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5158031990 con 0x7f515c0b7530 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f5153fff640 1 -- 172.21.15.1:0/395341869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158031d20 con 0x7f515c0b7530 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 -- 172.21.15.1:0/395341869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b7530 msgr2=0x7f515c0d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 --2- 172.21.15.1:0/395341869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b7530 0x7f515c0d09b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f515c0cfc80 tx=0x7f515802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 -- 172.21.15.1:0/395341869 shutdown_connections 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 --2- 172.21.15.1:0/395341869 >> v2:172.21.15.3:3300/0 conn(0x7f515c0b6a40 0x7f515c06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 --2- 172.21.15.1:0/395341869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b7530 0x7f515c0d09b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.287+0000 7f516337a640 1 -- 172.21.15.1:0/395341869 >> 172.21.15.1:0/395341869 conn(0x7f515c0d1c80 msgr2=0x7f515c0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.288+0000 7f516337a640 1 -- 172.21.15.1:0/395341869 shutdown_connections 2024-03-24T10:34:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.288+0000 7f516337a640 1 -- 172.21.15.1:0/395341869 wait complete. 2024-03-24T10:34:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 Processor -- start 2024-03-24T10:34:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 -- start start 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f515c06cba0 0x7f515c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f515c0b7530 0x7f515c14cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f515c14b330 con 0x7f515c0b6a40 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f515c14b000 con 0x7f515c06cba0 2024-03-24T10:34:00.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f516337a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f515c0c6d50 con 0x7f515c0b7530 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f5160915640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f5160915640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45436/0 (socket says 172.21.15.1:45436) 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f5161116640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f515c06cba0 0x7f515c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.289+0000 7f5160915640 1 -- 172.21.15.1:0/3020507529 learned_addr learned my addr 172.21.15.1:0/3020507529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5161917640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f515c0b7530 0x7f515c14cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 -- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f515c0b7530 msgr2=0x7f515c14cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f515c0b7530 0x7f515c14cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 -- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f515c06cba0 msgr2=0x7f515c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f515c06cba0 0x7f515c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 -- 172.21.15.1:0/3020507529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5154009a70 con 0x7f515c0b6a40 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5160915640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f514c00c400 tx=0x7f514c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c005ce0 con 0x7f515c0b6a40 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514c005e40 con 0x7f515c0b6a40 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c0072d0 con 0x7f515c0b6a40 2024-03-24T10:34:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5158009400 con 0x7f515c0b6a40 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.290+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515c0c7310 con 0x7f515c0b6a40 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.291+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c06de20 con 0x7f515c0b6a40 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.292+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f514c0074f0 con 0x7f515c0b6a40 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.292+0000 7f5151ffb640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 0x7f514006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.292+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f514c0896c0 con 0x7f515c0b6a40 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.292+0000 7f5161116640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 0x7f514006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.292+0000 7f5161116640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 0x7f514006dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5158036000 tx=0x7f515803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.296+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f514c054190 con 0x7f515c0b6a40 2024-03-24T10:34:00.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.479+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f515c04f4e0 con 0x7f515c0b6a40 2024-03-24T10:34:00.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.480+0000 7f5151ffb640 1 -- 172.21.15.1:0/3020507529 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f514c057850 con 0x7f515c0b6a40 2024-03-24T10:34:00.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:00.482 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 msgr2=0x7f514006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 0x7f514006dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5158036000 tx=0x7f515803c040 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 msgr2=0x7f515c0c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f514c00c400 tx=0x7f514c00c800 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 shutdown_connections 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f514006b810 0x7f514006dcd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f515c0b7530 0x7f515c14cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f515c0b6a40 0x7f515c0c6390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 --2- 172.21.15.1:0/3020507529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f515c06cba0 0x7f515c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 >> 172.21.15.1:0/3020507529 conn(0x7f515c0d1c80 msgr2=0x7f515c0d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 shutdown_connections 2024-03-24T10:34:00.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.482+0000 7f516337a640 1 -- 172.21.15.1:0/3020507529 wait complete. 2024-03-24T10:34:00.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-24T10:34:00.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-24T10:34:00.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-24T10:34:00.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-24T10:34:00.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-24T10:34:00.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-24T10:34:00.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-24T10:34:00.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-24T10:34:00.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.669+0000 7f94b759e640 1 Processor -- start 2024-03-24T10:34:00.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.669+0000 7fa2ff5ef640 1 Processor -- start 2024-03-24T10:34:00.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.669+0000 7f94b759e640 1 -- start start 2024-03-24T10:34:00.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.670+0000 7f248814f640 1 Processor -- start 2024-03-24T10:34:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.670+0000 7f9a9e3a1640 1 Processor -- start 2024-03-24T10:34:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.670+0000 7f9a9e3a1640 1 -- start start 2024-03-24T10:34:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.670+0000 7f248814f640 1 -- start start 2024-03-24T10:34:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f049159a640 1 Processor -- start 2024-03-24T10:34:00.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f049159a640 1 -- start start 2024-03-24T10:34:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.670+0000 7fa2ff5ef640 1 -- start start 2024-03-24T10:34:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b759e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94b8071aa0 0x7f94b8071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f87c0b01640 1 Processor -- start 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f87c0b01640 1 -- start start 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a9e3a1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9a981603e0 0x7f9a98167000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a9e3a1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a980d2d10 con 0x7f9a980d28a0 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a9e3a1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a980cd5d0 con 0x7f9a9815c350 2024-03-24T10:34:00.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a9e3a1640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9a980cca30 con 0x7f9a981603e0 2024-03-24T10:34:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7fa2ff5ef640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 0x7fa2f806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a97fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9a9815c350 0x7f9a980cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55266/0 (socket says 172.21.15.1:55266) 2024-03-24T10:34:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a97fff640 1 -- 172.21.15.1:0/1492468615 learned_addr learned my addr 172.21.15.1:0/1492468615 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:00.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f248814f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2478007330 0x7f247804f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f248814f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24780e7b30 con 0x7f24780e7750 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f248814f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2478007840 con 0x7f247800b0a0 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f248814f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2478007b20 con 0x7f2478007330 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f9a9c93e640 1 --2- 172.21.15.1:0/1492468615 >> v2:172.21.15.3:3300/0 conn(0x7f9a981603e0 0x7f9a98167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b759e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b806df50 con 0x7f94b806d500 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b759e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f94b8072470 con 0x7f94b8071aa0 2024-03-24T10:34:00.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b759e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f94b8140e00 con 0x7f94b80d0500 2024-03-24T10:34:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f55eac89640 1 Processor -- start 2024-03-24T10:34:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7ff9b491f640 1 Processor -- start 2024-03-24T10:34:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b659c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b659c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47880/0 (socket says 172.21.15.1:47880) 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.671+0000 7f94b659c640 1 -- 172.21.15.1:0/4103729155 learned_addr learned my addr 172.21.15.1:0/4103729155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f94b659c640 1 -- 172.21.15.1:0/4103729155 >> v2:172.21.15.1:3300/0 conn(0x7f94b8071aa0 msgr2=0x7f94b8071f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f94b659c640 1 --2- 172.21.15.1:0/4103729155 >> v2:172.21.15.1:3300/0 conn(0x7f94b8071aa0 0x7f94b8071f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f94b659c640 1 -- 172.21.15.1:0/4103729155 >> v1:172.21.15.1:6789/0 conn(0x7f94b806d500 legacy=0x7f94b8140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f94b659c640 1 -- 172.21.15.1:0/4103729155 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b81410e0 con 0x7f94b80d0500 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2ff5ef640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa2f806ce50 0x7fa2f81629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2ff5ef640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f80b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2ff5ef640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f80b3f40 con 0x7fa2f80cece0 2024-03-24T10:34:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f24866ec640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2478007330 0x7f247804f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f2485eeb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24780e7750 0x7f24780489c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41492/0 (socket says 172.21.15.1:41492) 2024-03-24T10:34:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f2485eeb640 1 -- 172.21.15.1:0/488652207 learned_addr learned my addr 172.21.15.1:0/488652207 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:00.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f049159a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f048c0cf4a0 0x7f048c0cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f049159a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c166010 0x7f048c1664a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f049159a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048c0b3a30 con 0x7f048c067170 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f049159a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f048c0b5dc0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f049159a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f048c162a40 con 0x7f048c166010 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2ff5ef640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f80b3950 con 0x7fa2f806ce50 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656876572 0 0) 0x7f24780e7b30 con 0x7f24780e7750 2024-03-24T10:34:00.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f246c003520 con 0x7f24780e7750 2024-03-24T10:34:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2ff5ef640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f8163430 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625679097 0 0) 0x7f9a88001110 con 0x7f9a980d28a0 2024-03-24T10:34:00.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a80003520 con 0x7f9a980d28a0 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2fd38b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 0x7fa2f806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2fd38b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 0x7fa2f806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33360/0 (socket says 172.21.15.1:33360) 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2fd38b640 1 -- 172.21.15.1:0/3756498302 learned_addr learned my addr 172.21.15.1:0/3756498302 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/3756498302 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f80b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f048affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f048c0cf4a0 0x7f048c0cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f048affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f048c0cf4a0 0x7f048c0cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33370/0 (socket says 172.21.15.1:33370) 2024-03-24T10:34:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7f048affd640 1 -- 172.21.15.1:0/1376182454 learned_addr learned my addr 172.21.15.1:0/1376182454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/3756498302 >> v2:172.21.15.3:3300/0 conn(0x7fa2f806ce50 msgr2=0x7fa2f81629e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:34:00.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/3756498302 >> v2:172.21.15.3:3300/0 conn(0x7fa2f806ce50 0x7fa2f81629e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f048affd640 1 -- 172.21.15.1:0/1376182454 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c166010 msgr2=0x7f048c1664a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f048affd640 1 --2- 172.21.15.1:0/1376182454 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c166010 0x7f048c1664a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f048affd640 1 -- 172.21.15.1:0/1376182454 >> v1:172.21.15.3:6789/0 conn(0x7f048c067170 legacy=0x7f048c0b3320 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:00.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f048affd640 1 -- 172.21.15.1:0/1376182454 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c0cfdc0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/3756498302 >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 msgr2=0x7fa2f806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/3756498302 >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 0x7fa2f806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/3756498302 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f8163710 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87c0b01640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87bc15dec0 0x7f87bc1624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87c0b01640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f87bc0ce940 0x7f87bc06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87c0b01640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc06e780 con 0x7f87bc06e330 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87c0b01640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc0b2bd0 con 0x7f87bc15dec0 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87c0b01640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc1629e0 con 0x7f87bc0ce940 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87b9d9b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f87bc0ce940 0x7f87bc06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87b9d9b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f87bc0ce940 0x7f87bc06ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49616/0 (socket says 172.21.15.1:49616) 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f87b9d9b640 1 -- 172.21.15.1:0/3863457009 learned_addr learned my addr 172.21.15.1:0/3863457009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f55eac89640 1 -- start start 2024-03-24T10:34:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.672+0000 7ff9b491f640 1 -- start start 2024-03-24T10:34:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9b491f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9a8009d90 0x7ff9a80e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9b491f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9b491f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a8051e70 con 0x7ff9a8051680 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9b491f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a80e2730 con 0x7ff9a8009d90 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9b491f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a8052150 con 0x7ff9a80e3480 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9affff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9af7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9a8009d90 0x7ff9a80e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9affff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47920/0 (socket says 172.21.15.1:47920) 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7ff9affff640 1 -- 172.21.15.1:0/3097467092 learned_addr learned my addr 172.21.15.1:0/3097467092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3663060812 0 0) 0x7f9a980cd5d0 con 0x7f9a9815c350 2024-03-24T10:34:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f94b659c640 1 --2- 172.21.15.1:0/4103729155 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009870 tx=0x7f94a802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4be71bbc34008ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/3756498302 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f80b43d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80cc490 tx=0x7fa2f002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9fb2252452aef4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a88001110 con 0x7f9a9815c350 2024-03-24T10:34:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f9a9c93e640 1 -- 172.21.15.1:0/1492468615 >> v1:172.21.15.3:6789/0 conn(0x7f9a9815c350 legacy=0x7f9a980cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f9a9c93e640 1 -- 172.21.15.1:0/1492468615 >> v1:172.21.15.1:6789/0 conn(0x7f9a980d28a0 legacy=0x7f9a9815fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f9a9c93e640 1 -- 172.21.15.1:0/1492468615 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a980ccca0 con 0x7f9a981603e0 2024-03-24T10:34:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f87ba59c640 1 --2- 172.21.15.1:0/3863457009 >> v2:172.21.15.1:3300/0 conn(0x7f87bc15dec0 0x7f87bc1624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f94b559a640 1 -- 172.21.15.1:0/4103729155 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8030860 con 0x7f94b80d0500 2024-03-24T10:34:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7fa2effff640 1 -- 172.21.15.1:0/3756498302 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f002cc00 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7fa2effff640 1 -- 172.21.15.1:0/3756498302 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f002cd60 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7fa2effff640 1 -- 172.21.15.1:0/3756498302 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0034660 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f55eac89640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e4144a50 0x7f55e406c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f87b9d9b640 1 -- 172.21.15.1:0/3863457009 >> v2:172.21.15.1:3300/0 conn(0x7f87bc15dec0 msgr2=0x7f87bc1624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f87b9d9b640 1 --2- 172.21.15.1:0/3863457009 >> v2:172.21.15.1:3300/0 conn(0x7f87bc15dec0 0x7f87bc1624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f87b9d9b640 1 -- 172.21.15.1:0/3863457009 >> v1:172.21.15.1:6789/0 conn(0x7f87bc06e330 legacy=0x7f87bc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f87b9d9b640 1 -- 172.21.15.1:0/3863457009 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc0ae3b0 con 0x7f87bc0ce940 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4135103517 0 0) 0x7f246c003520 con 0x7f24780e7750 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 >> v2:172.21.15.1:3300/0 conn(0x7f2478007330 msgr2=0x7f247804f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 --2- 172.21.15.1:0/488652207 >> v2:172.21.15.1:3300/0 conn(0x7f2478007330 0x7f247804f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 >> v1:172.21.15.3:6789/0 conn(0x7f247800b0a0 legacy=0x7f2478053f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2478053730 con 0x7f24780e7750 2024-03-24T10:34:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f247c001670 con 0x7f24780e7750 2024-03-24T10:34:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 >> v1:172.21.15.1:6789/0 conn(0x7f24780e7750 legacy=0x7f24780489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f246c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f246c004390 0x7f246c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f246c005060 0x7f246c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2478007b20 con 0x7f246c003970 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f246c00bfb0 con 0x7f246c004390 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f246c00c170 con 0x7f246c005060 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.673+0000 7f24866ec640 1 --2- 172.21.15.1:0/488652207 >> v2:172.21.15.1:3300/0 conn(0x7f2478007330 0x7f247804f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f2485eeb640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f246c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f24856ea640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f246c004390 0x7f246c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f55eac89640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55e406d4d0 0x7f55e4070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55eac89640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e4144e30 con 0x7f55e40cf2d0 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55e9486640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f55e40cf2d0 0x7f55e406cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41530/0 (socket says 172.21.15.1:41530) 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55e9486640 1 -- 172.21.15.1:0/2497704917 learned_addr learned my addr 172.21.15.1:0/2497704917 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55eac89640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f55e4072050 con 0x7f55e406d4d0 2024-03-24T10:34:00.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f2485eeb640 1 -- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f246c005060 msgr2=0x7f246c00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f55e40cee40 con 0x7f55e4144a50 2024-03-24T10:34:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55e9c87640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e4144a50 0x7f55e406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f048affd640 1 --2- 172.21.15.1:0/1376182454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c0cf4a0 0x7f048c0cf880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0480009870 tx=0x7f048002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e8bee418f9366d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f0489ffb640 1 -- 172.21.15.1:0/1376182454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480031870 con 0x7f048c0cf4a0 2024-03-24T10:34:00.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f24866ec640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f246c005060 0x7f246c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f0489ffb640 1 -- 172.21.15.1:0/1376182454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04800319d0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f2485eeb640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f246c005060 0x7f246c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f0489ffb640 1 -- 172.21.15.1:0/1376182454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480031d60 con 0x7f048c0cf4a0 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7ff9affff640 1 -- 172.21.15.1:0/3097467092 >> v2:172.21.15.1:3300/0 conn(0x7ff9a8009d90 msgr2=0x7ff9a80e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7ff9affff640 1 --2- 172.21.15.1:0/3097467092 >> v2:172.21.15.1:3300/0 conn(0x7ff9a8009d90 0x7ff9a80e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7ff9affff640 1 -- 172.21.15.1:0/3097467092 >> v1:172.21.15.1:6789/0 conn(0x7ff9a8051680 legacy=0x7ff9a800a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7ff9affff640 1 -- 172.21.15.1:0/3097467092 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a80e2c80 con 0x7ff9a80e3480 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f2485eeb640 1 -- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f246c004390 msgr2=0x7f246c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f2485eeb640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f246c004390 0x7f246c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f2485eeb640 1 -- 172.21.15.1:0/488652207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2478053730 con 0x7f246c003970 2024-03-24T10:34:00.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55ea488640 1 --2- 172.21.15.1:0/2497704917 >> v2:172.21.15.3:3300/0 conn(0x7f55e406d4d0 0x7f55e4070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f24866ec640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f246c005060 0x7f246c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:00.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f24866ec640 1 -- 172.21.15.1:0/488652207 reap_dead start 2024-03-24T10:34:00.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f9a9c93e640 1 --2- 172.21.15.1:0/1492468615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a981603e0 0x7f9a98167000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009d90 tx=0x7f9a9002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e360bb7139e4bd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/3756498302 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 msgr2=0x7fa2f80b43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 --2- 172.21.15.1:0/3756498302 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f80b43d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80cc490 tx=0x7fa2f002be50 comp rx=0 tx=0).stop 2024-03-24T10:34:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/3756498302 shutdown_connections 2024-03-24T10:34:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55ea488640 1 -- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e4144a50 msgr2=0x7f55e406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:00.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55ea488640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e4144a50 0x7f55e406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55ea488640 1 -- 172.21.15.1:0/2497704917 >> v1:172.21.15.1:6789/0 conn(0x7f55e40cf2d0 legacy=0x7f55e406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55ea488640 1 -- 172.21.15.1:0/2497704917 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e4072dd0 con 0x7f55e406d4d0 2024-03-24T10:34:00.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7f55e8c85640 1 -- 172.21.15.1:0/2497704917 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665195288 0 0) 0x7f55e0001110 con 0x7f55e40cf2d0 2024-03-24T10:34:00.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9002cc60 con 0x7f9a981603e0 2024-03-24T10:34:00.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9002cdc0 con 0x7f9a981603e0 2024-03-24T10:34:00.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f94b559a640 1 -- 172.21.15.1:0/4103729155 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a80309c0 con 0x7f94b80d0500 2024-03-24T10:34:00.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.674+0000 7f94b559a640 1 -- 172.21.15.1:0/4103729155 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8030d50 con 0x7f94b80d0500 2024-03-24T10:34:00.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f2485eeb640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f246c003d70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c370 tx=0x7f246c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150b387d5326e404 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 -- 172.21.15.1:0/1376182454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c0cf4a0 msgr2=0x7f048c0cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 --2- 172.21.15.1:0/1376182454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c0cf4a0 0x7f048c0cf880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0480009870 tx=0x7f048002b910 comp rx=0 tx=0).stop 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 -- 172.21.15.1:0/4103729155 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 msgr2=0x7f94b8143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 --2- 172.21.15.1:0/4103729155 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009870 tx=0x7f94a802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 -- 172.21.15.1:0/4103729155 shutdown_connections 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 --2- 172.21.15.1:0/4103729155 >> v2:172.21.15.1:3300/0 conn(0x7f94b8071aa0 0x7f94b8071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f87b9d9b640 1 --2- 172.21.15.1:0/3863457009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc06ddf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87bc070820 tx=0x7f87b002b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a2981356d3adf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 --2- 172.21.15.1:0/3756498302 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f80b43d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 --2- 172.21.15.1:0/4103729155 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b80d0500 0x7f94b8143620 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009870 tx=0x7f94a802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f94b759e640 1 -- 172.21.15.1:0/4103729155 >> 172.21.15.1:0/4103729155 conn(0x7f94b80cfa10 msgr2=0x7f94b80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1492468615 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90034660 con 0x7f9a981603e0 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7ff9affff640 1 --2- 172.21.15.1:0/3097467092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009870 tx=0x7ff9a402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8cb0d1bf28fed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7ff9ae7fc640 1 -- 172.21.15.1:0/3097467092 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4031640 con 0x7ff9a80e3480 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7ff9ae7fc640 1 -- 172.21.15.1:0/3097467092 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a40317a0 con 0x7ff9a80e3480 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 --2- 172.21.15.1:0/3756498302 >> v2:172.21.15.3:3300/0 conn(0x7fa2f806ce50 0x7fa2f81629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 --2- 172.21.15.1:0/3756498302 >> v2:172.21.15.1:3300/0 conn(0x7fa2f80cece0 0x7fa2f806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.675+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/3756498302 >> 172.21.15.1:0/3756498302 conn(0x7fa2f80b17a0 msgr2=0x7fa2f80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 -- 172.21.15.1:0/1376182454 shutdown_connections 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 --2- 172.21.15.1:0/1376182454 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c166010 0x7f048c1664a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 --2- 172.21.15.1:0/1376182454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c0cf4a0 0x7f048c0cf880 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c02cba0 con 0x7f246c003970 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f247c02cd00 con 0x7f246c003970 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f2484ee9640 1 -- 172.21.15.1:0/488652207 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c005af0 con 0x7f246c003970 2024-03-24T10:34:00.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.676+0000 7f049159a640 1 -- 172.21.15.1:0/1376182454 >> 172.21.15.1:0/1376182454 conn(0x7f048c0d02d0 msgr2=0x7f048c0d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/1492468615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a981603e0 msgr2=0x7f9a98167000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 --2- 172.21.15.1:0/1492468615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a981603e0 0x7f9a98167000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009d90 tx=0x7f9a9002beb0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f94b759e640 1 -- 172.21.15.1:0/4103729155 shutdown_connections 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/3756498302 shutdown_connections 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/3756498302 wait complete. 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f94b759e640 1 -- 172.21.15.1:0/4103729155 wait complete. 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/1492468615 shutdown_connections 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 --2- 172.21.15.1:0/1492468615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a981603e0 0x7f9a98167000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/1492468615 >> 172.21.15.1:0/1492468615 conn(0x7f9a980d1d80 msgr2=0x7f9a980d2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 -- 172.21.15.1:0/3097467092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 msgr2=0x7ff9a80e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 --2- 172.21.15.1:0/3097467092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009870 tx=0x7ff9a402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/1492468615 shutdown_connections 2024-03-24T10:34:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/1492468615 wait complete. 2024-03-24T10:34:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 -- 172.21.15.1:0/3097467092 shutdown_connections 2024-03-24T10:34:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 --2- 172.21.15.1:0/3097467092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a80e3480 0x7ff9a80e3910 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 --2- 172.21.15.1:0/3097467092 >> v2:172.21.15.1:3300/0 conn(0x7ff9a8009d90 0x7ff9a80e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7ff9b491f640 1 -- 172.21.15.1:0/3097467092 >> 172.21.15.1:0/3097467092 conn(0x7ff9a80537a0 msgr2=0x7ff9a8053b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f55ea488640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e406d4d0 0x7f55e4070b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009d90 tx=0x7f55dc02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aa58b1c8a0b8160 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f55e8c85640 1 -- 172.21.15.1:0/2497704917 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc02cbd0 con 0x7f55e406d4d0 2024-03-24T10:34:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f55e8c85640 1 -- 172.21.15.1:0/2497704917 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55dc02cd30 con 0x7f55e406d4d0 2024-03-24T10:34:00.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f55e8c85640 1 -- 172.21.15.1:0/2497704917 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc02cea0 con 0x7f55e406d4d0 2024-03-24T10:34:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f049159a640 1 -- 172.21.15.1:0/1376182454 shutdown_connections 2024-03-24T10:34:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f049159a640 1 -- 172.21.15.1:0/1376182454 wait complete. 2024-03-24T10:34:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f87b959a640 1 -- 172.21.15.1:0/3863457009 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0030870 con 0x7f87bc0ce940 2024-03-24T10:34:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f87b959a640 1 -- 172.21.15.1:0/3863457009 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b00309d0 con 0x7f87bc0ce940 2024-03-24T10:34:00.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f87b959a640 1 -- 172.21.15.1:0/3863457009 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0030d60 con 0x7f87bc0ce940 2024-03-24T10:34:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f248814f640 1 -- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 msgr2=0x7f246c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.677+0000 7f248814f640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f246c003d70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c370 tx=0x7f246c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f248814f640 1 -- 172.21.15.1:0/488652207 shutdown_connections 2024-03-24T10:34:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f248814f640 1 --2- 172.21.15.1:0/488652207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f246c003d70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f248814f640 1 -- 172.21.15.1:0/488652207 >> 172.21.15.1:0/488652207 conn(0x7f247800a3f0 msgr2=0x7f247800a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 -- 172.21.15.1:0/3863457009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 msgr2=0x7f87bc06ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 --2- 172.21.15.1:0/3863457009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc06ddf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87bc070820 tx=0x7f87b002b9b0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 -- 172.21.15.1:0/3863457009 shutdown_connections 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 --2- 172.21.15.1:0/3863457009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc06ddf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 --2- 172.21.15.1:0/3863457009 >> v2:172.21.15.1:3300/0 conn(0x7f87bc15dec0 0x7f87bc1624a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 -- 172.21.15.1:0/3863457009 >> 172.21.15.1:0/3863457009 conn(0x7f87bc06b0a0 msgr2=0x7f87bc06b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e406d4d0 msgr2=0x7f55e4070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 -- 172.21.15.1:0/3863457009 shutdown_connections 2024-03-24T10:34:00.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e406d4d0 0x7f55e4070b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009d90 tx=0x7f55dc02be20 comp rx=0 tx=0).stop 2024-03-24T10:34:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f87c0b01640 1 -- 172.21.15.1:0/3863457009 wait complete. 2024-03-24T10:34:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7ff9b491f640 1 -- 172.21.15.1:0/3097467092 shutdown_connections 2024-03-24T10:34:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 shutdown_connections 2024-03-24T10:34:00.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e406d4d0 0x7f55e4070b00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f248814f640 1 -- 172.21.15.1:0/488652207 shutdown_connections 2024-03-24T10:34:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 --2- 172.21.15.1:0/2497704917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e4144a50 0x7f55e406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 >> 172.21.15.1:0/2497704917 conn(0x7f55e40b16a0 msgr2=0x7f55e40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7fa2ff5ef640 1 Processor -- start 2024-03-24T10:34:00.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f049159a640 1 Processor -- start 2024-03-24T10:34:00.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7fa2ff5ef640 1 -- start start 2024-03-24T10:34:00.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f9a9e3a1640 1 Processor -- start 2024-03-24T10:34:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f9a9e3a1640 1 -- start start 2024-03-24T10:34:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f94b759e640 1 Processor -- start 2024-03-24T10:34:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f94b759e640 1 -- start start 2024-03-24T10:34:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f87c0b01640 1 Processor -- start 2024-03-24T10:34:00.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f87c0b01640 1 -- start start 2024-03-24T10:34:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7f248814f640 1 -- 172.21.15.1:0/488652207 wait complete. 2024-03-24T10:34:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 shutdown_connections 2024-03-24T10:34:00.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f55eac89640 1 -- 172.21.15.1:0/2497704917 wait complete. 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.678+0000 7ff9b491f640 1 -- 172.21.15.1:0/3097467092 wait complete. 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7ff9b491f640 1 Processor -- start 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2f806ce50 0x7fa2f815ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f80cece0 0x7fa2f8164c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fcb8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f80cece0 0x7fa2f8164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fcb8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f80cece0 0x7fa2f8164c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33418/0 (socket says 172.21.15.1:33418) 2024-03-24T10:34:00.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fd38b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2f806ce50 0x7fa2f815ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fd38b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2f806ce50 0x7fa2f815ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49642/0 (socket says 172.21.15.1:49642) 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fcb8a640 1 -- 172.21.15.1:0/1686599179 learned_addr learned my addr 172.21.15.1:0/1686599179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f8161720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f815fa80 con 0x7fa2f80cece0 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f8161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f815ee70 con 0x7fa2f806ce50 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f815f150 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.679+0000 7f049159a640 1 -- start start 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c067170 0x7f048c14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 0x7f048c06f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c166010 0x7f048c15d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f048c06e680 con 0x7f048c166010 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f048c06eaa0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f049159a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f048c0b1980 con 0x7f048c067170 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f048a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 0x7f048c06f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f048a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 0x7f048c06f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49650/0 (socket says 172.21.15.1:49650) 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f048a7fc640 1 -- 172.21.15.1:0/4242025350 learned_addr learned my addr 172.21.15.1:0/4242025350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 0x7f94b814c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f248814f640 1 Processor -- start 2024-03-24T10:34:00.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7f248814f640 1 -- start start 2024-03-24T10:34:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048b7fe640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c166010 0x7f048c15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b8071aa0 0x7f94b816a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94b81353e0 0x7f94b80cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 0x7f9a980b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a9815c350 0x7f9a9815d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9a981603e0 0x7f9a9815dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9815b4d0 con 0x7f9a9815c350 2024-03-24T10:34:00.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9a980af2c0 con 0x7f9a980d28a0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.680+0000 7ff9b491f640 1 -- start start 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a9e3a1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9a981533e0 con 0x7f9a981603e0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f94b816a020 con 0x7f94b81353e0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f94b814ce10 con 0x7f94b806d500 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b759e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f94b8169410 con 0x7f94b8071aa0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 0x7f94b814c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b5d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b8071aa0 0x7f94b816a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 0x7f94b814c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49656/0 (socket says 172.21.15.1:49656) 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f94b659c640 1 -- 172.21.15.1:0/2883007892 learned_addr learned my addr 172.21.15.1:0/2883007892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 0x7f9a980b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 0x7f9a980b5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49672/0 (socket says 172.21.15.1:49672) 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f9a97fff640 1 -- 172.21.15.1:0/2032395331 learned_addr learned my addr 172.21.15.1:0/2032395331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9a8009d90 0x7ff9a80ddf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 0x7ff9a804f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a804fec0 0x7ff9a80089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a8050350 con 0x7ff9a804fec0 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80de440 con 0x7ff9a8009d90 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7ff9b491f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a816aad0 con 0x7ff9a8051680 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2f806ce50 msgr2=0x7fa2f815ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2f806ce50 0x7fa2f815ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f80cece0 msgr2=0x7fa2f8164c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f80cece0 0x7fa2f8164c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f0009a70 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 -- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c067170 msgr2=0x7f048c14e880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f048c067170 0x7f048c14e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 -- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c166010 msgr2=0x7f048c15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f048c166010 0x7f048c15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04800094d0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.681+0000 7f048a7fc640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 0x7f048c06f2a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f048400b9e0 tx=0x7f048400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048400cba0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f94b659c640 1 -- 172.21.15.1:0/2883007892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b8071aa0 msgr2=0x7f94b816a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f94b659c640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94b8071aa0 0x7f94b816a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f94b659c640 1 -- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94b81353e0 msgr2=0x7f94b80cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f94b659c640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94b81353e0 0x7f94b80cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f94b659c640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a80094d0 con 0x7f94b806d500 2024-03-24T10:34:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a97fff640 1 -- 172.21.15.1:0/2032395331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9a981603e0 msgr2=0x7f9a9815dc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:34:00.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a977fe640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a9815c350 0x7f9a9815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9aeffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 0x7ff9a804f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a97fff640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9a981603e0 0x7f9a9815dc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a97fff640 1 -- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a9815c350 msgr2=0x7f9a9815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a97fff640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a9815c350 0x7f9a9815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a97fff640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a90009a70 con 0x7f9a980d28a0 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f049159a640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048c06dfc0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f049159a640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048c06f7e0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f24780de750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 0x7f24780e8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24780e8a40 0x7f247817c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f24780dec90 con 0x7f246c003970 2024-03-24T10:34:00.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f24780e48c0 con 0x7f24780e5960 2024-03-24T10:34:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f248814f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f24780e4ba0 con 0x7f24780e8a40 2024-03-24T10:34:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f9a977fe640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a9815c350 0x7f9a9815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f55eac89640 1 Processor -- start 2024-03-24T10:34:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f048400cd00 con 0x7f048c0cf4a0 2024-03-24T10:34:00.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484014680 con 0x7f048c0cf4a0 2024-03-24T10:34:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2fdb8c640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f8161720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0033000 tx=0x7fa2f0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87bc06e330 0x7f87bc1fc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 0x7f24780e8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f55eac89640 1 -- start start 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87bc15dec0 0x7f87bc1516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc15e2a0 con 0x7f87bc06e330 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1f84a0 con 0x7f87bc0ce940 2024-03-24T10:34:00.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f87c0b01640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc15e560 con 0x7f87bc15dec0 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 0x7f24780e8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49692/0 (socket says 172.21.15.1:49692) 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 -- 172.21.15.1:0/2147248566 learned_addr learned my addr 172.21.15.1:0/2147248566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9aeffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 0x7ff9a804f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47968/0 (socket says 172.21.15.1:47968) 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9aeffd640 1 -- 172.21.15.1:0/260283780 learned_addr learned my addr 172.21.15.1:0/260283780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0040450 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f00405b0 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0040980 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f815f430 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9affff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a804fec0 0x7ff9a80089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9af7fe640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9a8009d90 0x7ff9a80ddf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24866ec640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24780e8a40 0x7f247817c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f8165150 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 -- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24780e8a40 msgr2=0x7f247817c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24780e8a40 0x7f247817c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f24856ea640 1 -- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 msgr2=0x7f24780de750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7f2485eeb640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f24780de750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9aeffd640 1 -- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9a8009d90 msgr2=0x7ff9a80ddf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.682+0000 7ff9aeffd640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9a8009d90 0x7ff9a80ddf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9aeffd640 1 -- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a804fec0 msgr2=0x7ff9a80089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9aeffd640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a804fec0 0x7ff9a80089d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9aeffd640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a40094d0 con 0x7ff9a8051680 2024-03-24T10:34:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9affff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a804fec0 0x7ff9a80089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24856ea640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f24780de750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24856ea640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24780e4e80 con 0x7f24780e5960 2024-03-24T10:34:00.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f2485eeb640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f246c003970 0x7f24780de750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87b9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87b9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49696/0 (socket says 172.21.15.1:49696) 2024-03-24T10:34:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87b9d9b640 1 -- 172.21.15.1:0/1729343727 learned_addr learned my addr 172.21.15.1:0/1729343727 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87ba59c640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87bc06e330 0x7f87bc1fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87bc15dec0 0x7f87bc1516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24856ea640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 0x7f24780e8500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2470007ae0 tx=0x7f247000e8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9aeffd640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 0x7ff9a804f980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff9b007aa50 tx=0x7ff9b00695d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f9a97fff640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 0x7f9a980b5730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c430 tx=0x7f9a8c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f55eac89640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e406d4d0 0x7f55e4131c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 -- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 msgr2=0x7f87bc155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87bc0ce940 0x7f87bc155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 -- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87bc06e330 msgr2=0x7f87bc1fc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87bc06e330 0x7f87bc1fc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f87bad9d640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b00094d0 con 0x7f87bc15dec0 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c005ce0 con 0x7f9a980d28a0 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c005e40 con 0x7f9a980d28a0 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0072f0 con 0x7f9a980d28a0 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9815b6e0 con 0x7f9a980d28a0 2024-03-24T10:34:00.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470003b30 con 0x7f24780e5960 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2470003c90 con 0x7f24780e5960 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470004050 con 0x7f24780e5960 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f248814f640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24780e5160 con 0x7f24780e5960 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f248814f640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24780e4550 con 0x7f24780e5960 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b00732a0 con 0x7ff9a8051680 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b0070040 con 0x7ff9a8051680 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b00736a0 con 0x7ff9a8051680 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.683+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a804cff0 con 0x7ff9a8051680 2024-03-24T10:34:00.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a8050820 con 0x7ff9a8051680 2024-03-24T10:34:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2f815ce50 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f04840147e0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f94b659c640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 0x7f94b814c8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94a802bf30 tx=0x7f94a8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f04737fe640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f047806b8e0 0x7f047806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55eac89640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e40cf2d0 0x7f55e4140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55eac89640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e4144a50 0x7f55e4132770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55eac89640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f55e412f640 con 0x7f55e40cf2d0 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55eac89640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f55e412f920 con 0x7f55e4144a50 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55eac89640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f55e412fc00 con 0x7f55e406d4d0 2024-03-24T10:34:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa2f0040ae0 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f049159a640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f046c001030 con 0x7f048c0cf4a0 2024-03-24T10:34:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55ea488640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e4144a50 0x7f55e4132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9a8006070 con 0x7ff9a8051680 2024-03-24T10:34:00.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55ea488640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e4144a50 0x7f55e4132770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49708/0 (socket says 172.21.15.1:49708) 2024-03-24T10:34:00.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55ea488640 1 -- 172.21.15.1:0/1375338321 learned_addr learned my addr 172.21.15.1:0/1375338321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:00.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55e9486640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e40cf2d0 0x7f55e4140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f87bad9d640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87bc15dec0 0x7f87bc1516a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c730 tx=0x7f87b400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f55e9c87640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e406d4d0 0x7f55e4131c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.684+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a980af4d0 con 0x7f9a980d28a0 2024-03-24T10:34:00.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9a980cd260 con 0x7f9a980d28a0 2024-03-24T10:34:00.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2478053420 con 0x7f24780e5960 2024-03-24T10:34:00.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7fa2edffb640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa2dc06b840 0x7fa2dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4006d50 con 0x7f87bc15dec0 2024-03-24T10:34:00.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.685+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc15e840 con 0x7f87bc15dec0 2024-03-24T10:34:00.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc15ee00 con 0x7f87bc15dec0 2024-03-24T10:34:00.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa2f00acb70 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4003970 con 0x7f87bc15dec0 2024-03-24T10:34:00.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9c87640 1 -- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e4144a50 msgr2=0x7f55e4132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9c87640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55e4144a50 0x7f55e4132770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9c87640 1 -- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e40cf2d0 msgr2=0x7f55e4140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9c87640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e40cf2d0 0x7f55e4140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9c87640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc009a70 con 0x7f55e406d4d0 2024-03-24T10:34:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f55e9486640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e40cf2d0 0x7f55e4140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:00.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4003d60 con 0x7f87bc15dec0 2024-03-24T10:34:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7fa2edffb640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fa2dc073a60 0x7fa2dc075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2dc076530 con 0x7fa2dc073a60 2024-03-24T10:34:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa2f00b12a0 con 0x7fa2f80cf0c0 2024-03-24T10:34:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8030810 con 0x7f94b806d500 2024-03-24T10:34:00.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a8043400 con 0x7f94b806d500 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8043770 con 0x7f94b806d500 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b81696f0 con 0x7f94b806d500 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b8169cb0 con 0x7f94b806d500 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0484089ff0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f04737fe640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0478073b00 0x7f0478075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff9b0071770 con 0x7ff9a8051680 2024-03-24T10:34:00.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04780765d0 con 0x7f0478073b00 2024-03-24T10:34:00.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f048408f5a0 con 0x7f048c0cf4a0 2024-03-24T10:34:00.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.686+0000 7f048affd640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f047806b8e0 0x7f047806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f55e9c87640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e406d4d0 0x7f55e4131c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e870 tx=0x7f55d400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff99bfff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff99406b8e0 0x7ff99406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff9b0105180 con 0x7ff9a8051680 2024-03-24T10:34:00.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff9af7fe640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff99406b8e0 0x7ff99406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff99bfff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7ff994073b00 0x7ff994075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7fa2fcb8a640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fa2dc073a60 0x7fa2dc075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9940765d0 con 0x7ff994073b00 2024-03-24T10:34:00.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24700041d0 con 0x7f24780e5960 2024-03-24T10:34:00.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff9b0105530 con 0x7ff9a8051680 2024-03-24T10:34:00.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f048affd640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f047806b8e0 0x7f047806dda0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0480036000 tx=0x7f0480008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f949c001030 con 0x7f94b806d500 2024-03-24T10:34:00.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f87b40050d0 con 0x7f87bc15dec0 2024-03-24T10:34:00.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7f048b7fe640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0478073b00 0x7f0478075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7ff9affff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7ff994073b00 0x7ff994075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24767fc640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f246406b910 0x7f246406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f048b7fe640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0478073b00 0x7f0478075f20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7ff9af7fe640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff99406b8e0 0x7ff99406dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009a50 tx=0x7ff99c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87a0001030 con 0x7f87bc15dec0 2024-03-24T10:34:00.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87aaffd640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f879c06b810 0x7f879c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f247008ac60 con 0x7f24780e5960 2024-03-24T10:34:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f2485eeb640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f246406b910 0x7f246406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7ff9affff640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7ff994073b00 0x7ff994075f20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.687+0000 7fa2fd38b640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa2dc06b840 0x7fa2dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7fa2fd38b640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa2dc06b840 0x7fa2dc06dd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009bd0 tx=0x7fa2f4009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9a8c007450 con 0x7f9a980d28a0 2024-03-24T10:34:00.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f87b4089980 con 0x7f87bc15dec0 2024-03-24T10:34:00.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d400ee60 con 0x7f55e406d4d0 2024-03-24T10:34:00.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87ba59c640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f879c06b810 0x7f879c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d400c940 con 0x7f55e406d4d0 2024-03-24T10:34:00.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e412fee0 con 0x7f55e406d4d0 2024-03-24T10:34:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87aaffd640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f879c0739b0 0x7f879c075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f879c076480 con 0x7f879c0739b0 2024-03-24T10:34:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f87b4001690 con 0x7f87bc15dec0 2024-03-24T10:34:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a94ff9640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9a8406b810 0x7f9a8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9a8c089fd0 con 0x7f9a980d28a0 2024-03-24T10:34:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== osd.7 v2:172.21.15.3:6816/1606588586 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04780765d0 con 0x7f0478073b00 2024-03-24T10:34:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a94ff9640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7f9a84073a90 0x7f9a84075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a84076560 con 0x7f9a84073a90 2024-03-24T10:34:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d400ccd0 con 0x7f55e406d4d0 2024-03-24T10:34:00.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e40ae9f0 con 0x7f55e406d4d0 2024-03-24T10:34:00.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c08a380 con 0x7f9a980d28a0 2024-03-24T10:34:00.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f9a9c93e640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7f9a84073a90 0x7f9a84075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7fa2fcb8a640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fa2dc073a60 0x7fa2dc075e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== osd.5 v2:172.21.15.3:6824/3670894664 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9940765d0 con 0x7ff994073b00 2024-03-24T10:34:00.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f87ba59c640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f879c06b810 0x7f879c06dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0079a0 tx=0x7f87ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f94a80438d0 con 0x7f94b806d500 2024-03-24T10:34:00.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24767fc640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7f2464073b90 0x7f2464075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2464076660 con 0x7f2464073b90 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f247008b010 con 0x7f24780e5960 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.688+0000 7f24866ec640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7f2464073b90 0x7f2464075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f2485eeb640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f246406b910 0x7f246406ddd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f247c034000 tx=0x7f247c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.689+0000 7f24866ec640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7f2464073b90 0x7f2464075fb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f55d4013470 con 0x7f55e406d4d0 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f94a6ffd640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f949806b910 0x7f949806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f94a803e070 con 0x7f94b806d500 2024-03-24T10:34:00.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f55e40cead0 con 0x7f55e406d4d0 2024-03-24T10:34:00.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== osd.3 v2:172.21.15.1:6817/2415676065 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2464076660 con 0x7f2464073b90 2024-03-24T10:34:00.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f94a6ffd640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7f9498073b30 0x7f9498075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9498076600 con 0x7f9498073b30 2024-03-24T10:34:00.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.690+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f94a80addf0 con 0x7f94b806d500 2024-03-24T10:34:00.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55da7fc640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55c806b860 0x7f55c806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f87b9d9b640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f879c0739b0 0x7f879c075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f55d40898f0 con 0x7f55e406d4d0 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55e9486640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55c806b860 0x7f55c806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55da7fc640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7f55c8073a80 0x7f55c8075ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55c8076550 con 0x7f55c8073a80 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f55d4089bf0 con 0x7f55e406d4d0 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f94b6d9d640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7f9498073b30 0x7f9498075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f94b5d9b640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f949806b910 0x7f949806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55ea488640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7f55c8073a80 0x7f55c8075ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f94b6d9d640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7f9498073b30 0x7f9498075f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f55e9486640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55c806b860 0x7f55c806dd20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55e0007900 tx=0x7f55e0007870 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f87b9d9b640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f879c0739b0 0x7f879c075dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.691+0000 7f9a977fe640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9a8406b810 0x7f9a8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.692+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== osd.6 v2:172.21.15.3:6808/2701673417 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2dc076530 con 0x7fa2dc073a60 2024-03-24T10:34:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.692+0000 7f55ea488640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7f55c8073a80 0x7f55c8075ea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.692+0000 7f9a9c93e640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7f9a84073a90 0x7f9a84075eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.692+0000 7f9a977fe640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9a8406b810 0x7f9a8406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a9815f430 tx=0x7f9a88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.692+0000 7f94b5d9b640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f949806b910 0x7f949806ddd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94b80cf790 tx=0x7f94ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.693+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== osd.0 v2:172.21.15.1:6808/2235589886 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9498076600 con 0x7f9498073b30 2024-03-24T10:34:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.693+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== osd.1 v2:172.21.15.1:6800/3667481935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f879c076480 con 0x7f879c0739b0 2024-03-24T10:34:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.693+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== osd.2 v2:172.21.15.1:6816/1358018271 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a84076560 con 0x7f9a84073a90 2024-03-24T10:34:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.693+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== osd.4 v2:172.21.15.3:6800/3439594233 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55c8076550 con 0x7f55c8073a80 2024-03-24T10:34:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.721+0000 7f049159a640 1 -- 172.21.15.1:0/4242025350 --> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f046c002d30 con 0x7f0478073b00 2024-03-24T10:34:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.722+0000 7f04737fe640 1 -- 172.21.15.1:0/4242025350 <== osd.7 v2:172.21.15.3:6816/1606588586 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f046c002d30 con 0x7f0478073b00 2024-03-24T10:34:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.722+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 --> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff9a80e1510 con 0x7ff994073b00 2024-03-24T10:34:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0478073b00 msgr2=0x7f0478075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0478073b00 0x7f0478075f20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 --> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2478005700 con 0x7f2464073b90 2024-03-24T10:34:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7ff99bfff640 1 -- 172.21.15.1:0/260283780 <== osd.5 v2:172.21.15.3:6824/3670894664 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff9a4031a70 con 0x7ff994073b00 2024-03-24T10:34:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f047806b8e0 msgr2=0x7f047806dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f047806b8e0 0x7f047806dda0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0480036000 tx=0x7f0480008030 comp rx=0 tx=0).stop 2024-03-24T10:34:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 msgr2=0x7f048c06f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 --2- 172.21.15.1:0/4242025350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0cf4a0 0x7f048c06f2a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f048400b9e0 tx=0x7f048400bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f048b7fe640 1 -- 172.21.15.1:0/4242025350 reap_dead start 2024-03-24T10:34:00.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7ff994073b00 msgr2=0x7ff994075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7ff9b491f640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7ff994073b00 0x7ff994075f20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 shutdown_connections 2024-03-24T10:34:00.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.723+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 >> 172.21.15.1:0/4242025350 conn(0x7f048c0d02d0 msgr2=0x7f048c1646e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f24767fc640 1 -- 172.21.15.1:0/2147248566 <== osd.3 v2:172.21.15.1:6817/2415676065 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2478005700 con 0x7f2464073b90 2024-03-24T10:34:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff99406b8e0 msgr2=0x7ff99406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff99406b8e0 0x7ff99406dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009a50 tx=0x7ff99c008040 comp rx=0 tx=0).stop 2024-03-24T10:34:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 msgr2=0x7ff9a804f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 --2- 172.21.15.1:0/260283780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff9a8051680 0x7ff9a804f980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff9b007aa50 tx=0x7ff9b00695d0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 shutdown_connections 2024-03-24T10:34:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f04717fa640 1 -- 172.21.15.1:0/4242025350 wait complete. 2024-03-24T10:34:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9affff640 1 -- 172.21.15.1:0/260283780 reap_dead start 2024-03-24T10:34:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 shutdown_connections 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 >> 172.21.15.1:0/260283780 conn(0x7ff9a80537a0 msgr2=0x7ff9a8004e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7f2464073b90 msgr2=0x7f2464075fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7f2464073b90 0x7f2464075fb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f246406b910 msgr2=0x7f246406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f246406b910 0x7f246406ddd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f247c034000 tx=0x7f247c009030 comp rx=0 tx=0).stop 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 msgr2=0x7f24780e8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f2463fff640 1 --2- 172.21.15.1:0/2147248566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24780e5960 0x7f24780e8500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2470007ae0 tx=0x7f247000e8a0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7f24866ec640 1 -- 172.21.15.1:0/2147248566 reap_dead start 2024-03-24T10:34:00.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.724+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 shutdown_connections 2024-03-24T10:34:00.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 shutdown_connections 2024-03-24T10:34:00.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 >> 172.21.15.1:0/2147248566 conn(0x7f247800a3f0 msgr2=0x7f24780e6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7ff9b491f640 1 -- 172.21.15.1:0/260283780 wait complete. 2024-03-24T10:34:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 --> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f55e4072af0 con 0x7f55c8073a80 2024-03-24T10:34:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7fa2ff5ef640 1 -- 172.21.15.1:0/1686599179 --> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa2f815ce50 con 0x7fa2dc073a60 2024-03-24T10:34:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 shutdown_connections 2024-03-24T10:34:00.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f2463fff640 1 -- 172.21.15.1:0/2147248566 wait complete. 2024-03-24T10:34:00.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7fa2edffb640 1 -- 172.21.15.1:0/1686599179 <== osd.6 v2:172.21.15.3:6808/2701673417 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa2f815ce50 con 0x7fa2dc073a60 2024-03-24T10:34:00.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.725+0000 7f55da7fc640 1 -- 172.21.15.1:0/1375338321 <== osd.4 v2:172.21.15.3:6800/3439594233 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f55dc02cb90 con 0x7f55c8073a80 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fa2dc073a60 msgr2=0x7fa2dc075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fa2dc073a60 0x7fa2dc075e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa2dc06b840 msgr2=0x7fa2dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa2dc06b840 0x7fa2dc06dd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009bd0 tx=0x7fa2f4009240 comp rx=0 tx=0).stop 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 msgr2=0x7fa2f8161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 --2- 172.21.15.1:0/1686599179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa2f80cf0c0 0x7fa2f8161720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0033000 tx=0x7fa2f0004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2fdb8c640 1 -- 172.21.15.1:0/1686599179 reap_dead start 2024-03-24T10:34:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 --> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f87a0002d30 con 0x7f879c0739b0 2024-03-24T10:34:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7f55c8073a80 msgr2=0x7f55c8075ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f55eac89640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7f55c8073a80 0x7f55c8075ea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 shutdown_connections 2024-03-24T10:34:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 >> 172.21.15.1:0/1686599179 conn(0x7fa2f80b17a0 msgr2=0x7fa2f80af710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55c806b860 msgr2=0x7f55c806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f55eac89640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55c806b860 0x7f55c806dd20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55e0007900 tx=0x7f55e0007870 comp rx=0 tx=0).stop 2024-03-24T10:34:00.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 --> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f949c002d30 con 0x7f9498073b30 2024-03-24T10:34:00.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.726+0000 7f94a6ffd640 1 -- 172.21.15.1:0/2883007892 <== osd.0 v2:172.21.15.1:6808/2235589886 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f949c002d30 con 0x7f9498073b30 2024-03-24T10:34:00.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 shutdown_connections 2024-03-24T10:34:00.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7fa2db7fe640 1 -- 172.21.15.1:0/1686599179 wait complete. 2024-03-24T10:34:00.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f87aaffd640 1 -- 172.21.15.1:0/1729343727 <== osd.1 v2:172.21.15.1:6800/3667481935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f87a0002d30 con 0x7f879c0739b0 2024-03-24T10:34:00.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 --> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9a9804f4e0 con 0x7f9a84073a90 2024-03-24T10:34:00.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7f9498073b30 msgr2=0x7f9498075f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f94b759e640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7f9498073b30 0x7f9498075f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e406d4d0 msgr2=0x7f55e4131c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 --2- 172.21.15.1:0/1375338321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55e406d4d0 0x7f55e4131c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e870 tx=0x7f55d400ed30 comp rx=0 tx=0).stop 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55ea488640 1 -- 172.21.15.1:0/1375338321 reap_dead start 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 shutdown_connections 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 >> 172.21.15.1:0/1375338321 conn(0x7f55e40b16a0 msgr2=0x7f55e415ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 shutdown_connections 2024-03-24T10:34:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f949806b910 msgr2=0x7f949806ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f94b759e640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f949806b910 0x7f949806ddd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94b80cf790 tx=0x7f94ac009040 comp rx=0 tx=0).stop 2024-03-24T10:34:00.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.727+0000 7f55eac89640 1 -- 172.21.15.1:0/1375338321 wait complete. 2024-03-24T10:34:00.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 msgr2=0x7f94b814c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a94ff9640 1 -- 172.21.15.1:0/2032395331 <== osd.2 v2:172.21.15.1:6816/1358018271 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9a9804f4e0 con 0x7f9a84073a90 2024-03-24T10:34:00.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b759e640 1 --2- 172.21.15.1:0/2883007892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94b806d500 0x7f94b814c8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94a802bf30 tx=0x7f94a8004730 comp rx=0 tx=0).stop 2024-03-24T10:34:00.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b6d9d640 1 -- 172.21.15.1:0/2883007892 reap_dead start 2024-03-24T10:34:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 shutdown_connections 2024-03-24T10:34:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 >> 172.21.15.1:0/2883007892 conn(0x7f94b80cfa10 msgr2=0x7f94b80b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f879c0739b0 msgr2=0x7f879c075dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f879c0739b0 0x7f879c075dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 shutdown_connections 2024-03-24T10:34:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f879c06b810 msgr2=0x7f879c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f879c06b810 0x7f879c06dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0079a0 tx=0x7f87ac008040 comp rx=0 tx=0).stop 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7f9a84073a90 msgr2=0x7f9a84075eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7f9a84073a90 0x7f9a84075eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9a8406b810 msgr2=0x7f9a8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9a8406b810 0x7f9a8406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a9815f430 tx=0x7f9a88009040 comp rx=0 tx=0).stop 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 msgr2=0x7f9a980b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9e3a1640 1 --2- 172.21.15.1:0/2032395331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a980d28a0 0x7f9a980b5730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c430 tx=0x7f9a8c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f9a9c93e640 1 -- 172.21.15.1:0/2032395331 reap_dead start 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87bc15dec0 msgr2=0x7f87bc1516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 --2- 172.21.15.1:0/1729343727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87bc15dec0 0x7f87bc1516a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c730 tx=0x7f87b400cbf0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87bad9d640 1 -- 172.21.15.1:0/1729343727 reap_dead start 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 shutdown_connections 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.728+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 >> 172.21.15.1:0/1729343727 conn(0x7f87bc06b0a0 msgr2=0x7f87bc0b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.729+0000 7f94b759e640 1 -- 172.21.15.1:0/2883007892 wait complete. 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.729+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 shutdown_connections 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.729+0000 7f87c0b01640 1 -- 172.21.15.1:0/1729343727 wait complete. 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.732+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 shutdown_connections 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.732+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 >> 172.21.15.1:0/2032395331 conn(0x7f9a980d1d80 msgr2=0x7f9a980cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.732+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 shutdown_connections 2024-03-24T10:34:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:00.733+0000 7f9a9e3a1640 1 -- 172.21.15.1:0/2032395331 wait complete. 2024-03-24T10:34:00.819 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:00.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T10:34:00.822 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:00.823 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T10:34:00.824 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:00.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T10:34:00.825 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:00.825 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T10:34:00.826 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:00.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T10:34:00.827 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:00.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T10:34:00.828 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:00.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T10:34:00.829 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:00.829 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T10:34:01.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.032+0000 7f7b4f343640 1 Processor -- start 2024-03-24T10:34:01.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.032+0000 7f7b4f343640 1 -- start start 2024-03-24T10:34:01.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4f343640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4f343640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b400e7b30 con 0x7f7b400e7750 2024-03-24T10:34:01.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4f343640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b40007840 con 0x7f7b4000b0a0 2024-03-24T10:34:01.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4f343640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7b40007b20 con 0x7f7b40007330 2024-03-24T10:34:01.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7fc35f5fd640 1 Processor -- start 2024-03-24T10:34:01.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4d8e0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4d8e0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48002/0 (socket says 172.21.15.1:48002) 2024-03-24T10:34:01.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7f7b4d8e0640 1 -- 172.21.15.1:0/2586070993 learned_addr learned my addr 172.21.15.1:0/2586070993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7fc35f5fd640 1 -- start start 2024-03-24T10:34:01.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.034+0000 7ff2f8c64640 1 Processor -- start 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35f5fd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc35815a3a0 0x7fc358071510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35f5fd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35db9a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc35806d8f0 0x7fc35815de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41534/0 (socket says 172.21.15.1:41534) 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35cb98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35cb98640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33506/0 (socket says 172.21.15.1:33506) 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35db9a640 1 -- 172.21.15.1:0/745356278 learned_addr learned my addr 172.21.15.1:0/745356278 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35815a780 con 0x7fc35806d8f0 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc35806dd10 con 0x7fc358071a50 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc3580cf120 con 0x7fc35815a3a0 2024-03-24T10:34:01.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fc35d399640 1 --2- 172.21.15.1:0/745356278 >> v2:172.21.15.3:3300/0 conn(0x7fc35815a3a0 0x7fc358071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.035+0000 7fade1409640 1 Processor -- start 2024-03-24T10:34:01.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.035+0000 7fade1409640 1 -- start start 2024-03-24T10:34:01.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fade1409640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faddc0cd340 0x7faddc06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fade1409640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faddc072cc0 con 0x7faddc06d060 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fade1409640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faddc0b34a0 con 0x7faddc14bf20 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7fade1409640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7faddc0b3780 con 0x7faddc0cd340 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.035+0000 7ff2f8c64640 1 -- start start 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7ff2f8c64640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f4072900 0x7ff2f40b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7ff2f8c64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2f40b3580 0x7ff2f406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7ff2f8c64640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2f414b4b0 con 0x7ff2f414b0d0 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7ff2f8c64640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f414bfc0 con 0x7ff2f40b3580 2024-03-24T10:34:01.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.036+0000 7ff2f8c64640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f40b4310 con 0x7ff2f4072900 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7faddb7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faddc06d060 0x7faddc06d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41542/0 (socket says 172.21.15.1:41542) 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7faddb7fe640 1 -- 172.21.15.1:0/1051338445 learned_addr learned my addr 172.21.15.1:0/1051338445 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fadda7fc640 1 --2- 172.21.15.1:0/1051338445 >> v2:172.21.15.1:3300/0 conn(0x7faddc0cd340 0x7faddc06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2f40b3580 0x7ff2f406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2f40b3580 0x7ff2f406d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33522/0 (socket says 172.21.15.1:33522) 2024-03-24T10:34:01.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/2607230461 learned_addr learned my addr 172.21.15.1:0/2607230461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f1d9b640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f4072900 0x7ff2f40b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7f7b4d8e0640 1 -- 172.21.15.1:0/2586070993 >> v1:172.21.15.3:6789/0 conn(0x7f7b4000b0a0 legacy=0x7f7b40053f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7f7b4d8e0640 1 -- 172.21.15.1:0/2586070993 >> v1:172.21.15.1:6789/0 conn(0x7f7b400e7750 legacy=0x7f7b400489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7f7b4d8e0640 1 -- 172.21.15.1:0/2586070993 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b40053730 con 0x7f7b40007330 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fc35d399640 1 -- 172.21.15.1:0/745356278 >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 msgr2=0x7fc35806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fc35d399640 1 --2- 172.21.15.1:0/745356278 >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fc35d399640 1 -- 172.21.15.1:0/745356278 >> v1:172.21.15.1:6789/0 conn(0x7fc35806d8f0 legacy=0x7fc35815de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fc35d399640 1 -- 172.21.15.1:0/745356278 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3580cf400 con 0x7fc35815a3a0 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fc35cb98640 1 --2- 172.21.15.1:0/745356278 >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456989449 0 0) 0x7faddc0b34a0 con 0x7faddc14bf20 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadc4003520 con 0x7faddc14bf20 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511313482 0 0) 0x7faddc072cc0 con 0x7faddc06d060 2024-03-24T10:34:01.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faddc0b34a0 con 0x7faddc06d060 2024-03-24T10:34:01.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7f06211c1640 1 Processor -- start 2024-03-24T10:34:01.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7f06211c1640 1 -- start start 2024-03-24T10:34:01.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/2607230461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f4072900 msgr2=0x7ff2f40b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f4072900 0x7ff2f40b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/2607230461 >> v1:172.21.15.1:6789/0 conn(0x7ff2f414b0d0 legacy=0x7ff2f40b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.037+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/2607230461 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f4072d60 con 0x7ff2f40b3580 2024-03-24T10:34:01.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7f7b4d8e0640 1 --2- 172.21.15.1:0/2586070993 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b380098a0 tx=0x7f7b3802ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af1b92c3ba1537d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7fc35d399640 1 --2- 172.21.15.1:0/745356278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35815a3a0 0x7fc358071510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc358073310 tx=0x7fc35002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8050608238d9671 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7fadda7fc640 1 -- 172.21.15.1:0/1051338445 >> v1:172.21.15.3:6789/0 conn(0x7faddc14bf20 legacy=0x7faddc0b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7fadda7fc640 1 -- 172.21.15.1:0/1051338445 >> v1:172.21.15.1:6789/0 conn(0x7faddc06d060 legacy=0x7faddc06d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.038+0000 7fadda7fc640 1 -- 172.21.15.1:0/1051338445 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc0b3a60 con 0x7faddc0cd340 2024-03-24T10:34:01.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f40b3580 0x7ff2f406d170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009e50 tx=0x7ff2ec02bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9073d4590b0c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7fc34ffff640 1 -- 172.21.15.1:0/745356278 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35002cc70 con 0x7fc35815a3a0 2024-03-24T10:34:01.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7ffb4e9e0640 1 Processor -- start 2024-03-24T10:34:01.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7fc34ffff640 1 -- 172.21.15.1:0/745356278 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc35002cdd0 con 0x7fc35815a3a0 2024-03-24T10:34:01.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f7b3ffff640 1 -- 172.21.15.1:0/2586070993 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802f820 con 0x7f7b40007330 2024-03-24T10:34:01.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7ff2f159a640 1 -- 172.21.15.1:0/2607230461 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec02c850 con 0x7ff2f40b3580 2024-03-24T10:34:01.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f7b3ffff640 1 -- 172.21.15.1:0/2586070993 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3802f980 con 0x7f7b40007330 2024-03-24T10:34:01.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f7b3ffff640 1 -- 172.21.15.1:0/2586070993 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3802fd10 con 0x7f7b40007330 2024-03-24T10:34:01.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06211c1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15f680 0x7f061c0ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f1b1493e640 1 Processor -- start 2024-03-24T10:34:01.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7fadda7fc640 1 --2- 172.21.15.1:0/1051338445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc0cd340 0x7faddc06cb20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009d90 tx=0x7fadd402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d61111b09e933c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06211c1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c0cda40 con 0x7f061c0d0710 2024-03-24T10:34:01.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f1b1493e640 1 -- start start 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06211c1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c0d2650 con 0x7f061c0b2a40 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06209c0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f061c0d0710 0x7f061c0b5300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41566/0 (socket says 172.21.15.1:41566) 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06209c0640 1 -- 172.21.15.1:0/3579924479 learned_addr learned my addr 172.21.15.1:0/3579924479 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f061c0d0ba0 con 0x7f061c15f680 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061bfff640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15f680 0x7f061c0ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35815a3a0 msgr2=0x7fc358071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/745356278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35815a3a0 0x7fc358071510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc358073310 tx=0x7fc35002bec0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.039+0000 7ffb4e9e0640 1 -- start start 2024-03-24T10:34:01.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd402ccf0 con 0x7faddc0cd340 2024-03-24T10:34:01.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7821c1f640 1 Processor -- start 2024-03-24T10:34:01.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ff2f159a640 1 -- 172.21.15.1:0/2607230461 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2ec02c9b0 con 0x7ff2f40b3580 2024-03-24T10:34:01.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ff2f159a640 1 -- 172.21.15.1:0/2607230461 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec02cd80 con 0x7ff2f40b3580 2024-03-24T10:34:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd402ce50 con 0x7faddc0cd340 2024-03-24T10:34:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fadd9ffb640 1 -- 172.21.15.1:0/1051338445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4034660 con 0x7faddc0cd340 2024-03-24T10:34:01.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002587820 0 0) 0x7f061c0cda40 con 0x7f061c0d0710 2024-03-24T10:34:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 -- 172.21.15.1:0/2586070993 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 msgr2=0x7f7b4004f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0604003520 con 0x7f061c0d0710 2024-03-24T10:34:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 --2- 172.21.15.1:0/2586070993 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b380098a0 tx=0x7f7b3802ba00 comp rx=0 tx=0).stop 2024-03-24T10:34:01.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ffb4e9e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb4815df60 0x7ffb480d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 -- 172.21.15.1:0/2586070993 shutdown_connections 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 --2- 172.21.15.1:0/2586070993 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b4004f610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 -- 172.21.15.1:0/2586070993 >> 172.21.15.1:0/2586070993 conn(0x7f7b4000a3f0 msgr2=0x7f7b4000a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7821c1f640 1 -- start start 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ffb4e9e0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb4815e340 0x7ffb480d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ffb4e9e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb480cfe50 con 0x7ffb480d12a0 2024-03-24T10:34:01.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ffb4e9e0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ffb480d49a0 con 0x7ffb4815df60 2024-03-24T10:34:01.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7ffb4e9e0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb480cedb0 con 0x7ffb4815e340 2024-03-24T10:34:01.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 shutdown_connections 2024-03-24T10:34:01.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/745356278 >> v2:172.21.15.1:3300/0 conn(0x7fc358071a50 0x7fc35806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/745356278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35815a3a0 0x7fc358071510 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc358073310 tx=0x7fc35002bec0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 >> 172.21.15.1:0/745356278 conn(0x7fc35806b0a0 msgr2=0x7fc35806b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1470772769 0 0) 0x7f0604003520 con 0x7f061c0d0710 2024-03-24T10:34:01.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f7b4f343640 1 -- 172.21.15.1:0/2586070993 shutdown_connections 2024-03-24T10:34:01.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b1493e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b101622d0 0x7f1b100cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b1493e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 0x7f1b1006b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ffb4e1df640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb480d12a0 0x7ffb480b4960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41576/0 (socket says 172.21.15.1:41576) 2024-03-24T10:34:01.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 shutdown_connections 2024-03-24T10:34:01.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7fc35f5fd640 1 -- 172.21.15.1:0/745356278 wait complete. 2024-03-24T10:34:01.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7fade1409640 1 -- 172.21.15.1:0/1051338445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc0cd340 msgr2=0x7faddc06cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7fade1409640 1 --2- 172.21.15.1:0/1051338445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc0cd340 0x7faddc06cb20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009d90 tx=0x7fadd402bf40 comp rx=0 tx=0).stop 2024-03-24T10:34:01.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 -- 172.21.15.1:0/2607230461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f40b3580 msgr2=0x7ff2f406d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f40b3580 0x7ff2f406d170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009e50 tx=0x7ff2ec02bd30 comp rx=0 tx=0).stop 2024-03-24T10:34:01.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b1493e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b10072db0 con 0x7f1b10072080 2024-03-24T10:34:01.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b1493e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1b10072bc0 con 0x7f1b101622d0 2024-03-24T10:34:01.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b1493e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1006ba70 con 0x7f1b1006b0a0 2024-03-24T10:34:01.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/2013779680 >> v2:172.21.15.1:3300/0 conn(0x7ffb4815df60 0x7ffb480d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ffb4e1df640 1 -- 172.21.15.1:0/2013779680 learned_addr learned my addr 172.21.15.1:0/2013779680 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:01.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ffb4d1dd640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb4815e340 0x7ffb480d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 -- 172.21.15.1:0/2607230461 shutdown_connections 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f4072900 0x7ff2f40b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/2607230461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f40b3580 0x7ff2f406d170 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7fade1409640 1 -- 172.21.15.1:0/1051338445 shutdown_connections 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7fade1409640 1 --2- 172.21.15.1:0/1051338445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc0cd340 0x7faddc06cb20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7fade1409640 1 -- 172.21.15.1:0/1051338445 >> 172.21.15.1:0/1051338445 conn(0x7faddc06b0a0 msgr2=0x7faddc06b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ff2f8c64640 1 -- 172.21.15.1:0/2607230461 >> 172.21.15.1:0/2607230461 conn(0x7ff2f406b0a0 msgr2=0x7ff2f406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b0dd9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 0x7f1b1006b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b0dd9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 0x7f1b1006b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48042/0 (socket says 172.21.15.1:48042) 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b0dd9b640 1 -- 172.21.15.1:0/2647110451 learned_addr learned my addr 172.21.15.1:0/2647110451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/2647110451 >> v2:172.21.15.1:3300/0 conn(0x7f1b101622d0 0x7f1b100cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7ffb4d9de640 1 -- 172.21.15.1:0/2013779680 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb4815e340 msgr2=0x7ffb480d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb4815e340 0x7ffb480d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f7821c1f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7814005ee0 0x7f7814006320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f7821c1f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7814054330 con 0x7f78140ead00 2024-03-24T10:34:01.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7814005ee0 0x7f7814006320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f782141e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7814006860 0x7f7814006cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55332/0 (socket says 172.21.15.1:55332) 2024-03-24T10:34:01.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f7b4f343640 1 -- 172.21.15.1:0/2586070993 wait complete. 2024-03-24T10:34:01.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7fade1409640 1 -- 172.21.15.1:0/1051338445 shutdown_connections 2024-03-24T10:34:01.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f1b0e59c640 1 -- 172.21.15.1:0/2647110451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 msgr2=0x7f1b1006b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/2647110451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 0x7f1b1006b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f1b0e59c640 1 -- 172.21.15.1:0/2647110451 >> v1:172.21.15.1:6789/0 conn(0x7f1b10072080 legacy=0x7f1b100724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f782141e640 1 -- 172.21.15.1:0/3104916770 learned_addr learned my addr 172.21.15.1:0/3104916770 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7814053720 con 0x7f7814006860 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7814053a00 con 0x7f7814005ee0 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477081574 0 0) 0x7f7814053720 con 0x7f7814006860 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f780c003520 con 0x7f7814006860 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.1 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2693753943 0 0) 0x7f780c003520 con 0x7f7814006860 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 >> v2:172.21.15.1:3300/0 conn(0x7f7814005ee0 msgr2=0x7f7814006320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 --2- 172.21.15.1:0/3104916770 >> v2:172.21.15.1:3300/0 conn(0x7f7814005ee0 0x7f7814006320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f1b0e59c640 1 -- 172.21.15.1:0/2647110451 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b100cfd80 con 0x7f1b101622d0 2024-03-24T10:34:01.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 >> v1:172.21.15.1:6789/0 conn(0x7f78140ead00 legacy=0x7f781404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7814053ce0 con 0x7f7814006860 2024-03-24T10:34:01.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.1 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f781c04f990 con 0x7f7814006860 2024-03-24T10:34:01.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 >> v1:172.21.15.3:6789/0 conn(0x7f7814006860 legacy=0x7f7814006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f7b4f343640 1 Processor -- start 2024-03-24T10:34:01.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f780c003970 0x7f780c003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 0x7f780c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f781b7fe640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f780c005060 0x7f780c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7814053a00 con 0x7f780c003970 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f780c00bfa0 con 0x7f780c005060 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f780c00c160 con 0x7f780c004380 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15f680 msgr2=0x7f061c0ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15f680 0x7f061c0ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 >> v1:172.21.15.3:6789/0 conn(0x7f061c0b2a40 legacy=0x7f061c0b4bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0d2930 con 0x7f061c0d0710 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f06140019f0 con 0x7f061c0d0710 2024-03-24T10:34:01.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.040+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 >> v1:172.21.15.1:6789/0 conn(0x7f061c0d0710 legacy=0x7f061c0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040039a0 0x7f0604003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06040047c0 0x7f0604004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0604009970 0x7f0604005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0d0ba0 con 0x7f0604009970 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0604014770 con 0x7f06040039a0 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0604014930 con 0x7f06040047c0 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040039a0 0x7f0604003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 -- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06040047c0 msgr2=0x7f0604004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f06040047c0 0x7f0604004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 -- 172.21.15.1:0/3579924479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0604009970 msgr2=0x7f0604005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0604009970 0x7f0604005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.041+0000 7f061bfff640 1 -- 172.21.15.1:0/3579924479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0d2930 con 0x7f06040039a0 2024-03-24T10:34:01.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7f061bfff640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040039a0 0x7f0604003da0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f061c071f10 tx=0x7f06040057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af472690934fd840 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 0x7f780c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f782141e640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f780c005060 0x7f780c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7ffb4d9de640 1 -- 172.21.15.1:0/2013779680 >> v1:172.21.15.1:6789/0 conn(0x7ffb480d12a0 legacy=0x7ffb480b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.042+0000 7ffb4d9de640 1 -- 172.21.15.1:0/2013779680 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb480cf090 con 0x7ffb4815df60 2024-03-24T10:34:01.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f7820c1d640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f780c003970 0x7f780c003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7fade1409640 1 -- 172.21.15.1:0/1051338445 wait complete. 2024-03-24T10:34:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 -- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f780c005060 msgr2=0x7f780c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f780c005060 0x7f780c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 -- 172.21.15.1:0/3104916770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f780c003970 msgr2=0x7f780c003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:01.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f780c003970 0x7f780c003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f781bfff640 1 -- 172.21.15.1:0/3104916770 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c003520 con 0x7f780c004380 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f782141e640 1 -- 172.21.15.1:0/3104916770 reap_dead start 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7fc35f5fd640 1 Processor -- start 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7fc35f5fd640 1 -- start start 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f7b4f343640 1 -- start start 2024-03-24T10:34:01.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061002c580 con 0x7f06040039a0 2024-03-24T10:34:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061002c6e0 con 0x7f06040039a0 2024-03-24T10:34:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f061affd640 1 -- 172.21.15.1:0/3579924479 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061002ca90 con 0x7f06040039a0 2024-03-24T10:34:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7ff2f8c64640 1 -- 172.21.15.1:0/2607230461 shutdown_connections 2024-03-24T10:34:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/2647110451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b101622d0 0x7f1b100cf840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b040097a0 tx=0x7f1b0402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaca2738b0bd5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.043+0000 7ff2f8c64640 1 -- 172.21.15.1:0/2607230461 wait complete. 2024-03-24T10:34:01.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb480d0d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffb440097a0 tx=0x7ffb4402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194d70ba71126104 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b400d2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 0x7f7b4004a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4004ef20 con 0x7f7b4000b0a0 2024-03-24T10:34:01.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7b40035430 con 0x7f7b400e7750 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4f343640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7b40034b00 con 0x7f7b40007330 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4c8de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4c8de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33574/0 (socket says 172.21.15.1:33574) 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4c8de640 1 -- 172.21.15.1:0/754545409 learned_addr learned my addr 172.21.15.1:0/754545409 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7fc35f5fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 0x7fc35814b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f781bfff640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 0x7f780c004860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba30 tx=0x7f780c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29c2a0b6aab82c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4d8e0640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 0x7f7b4004a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4d0df640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b400d2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040039a0 msgr2=0x7f0604003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f06211c1640 1 --2- 172.21.15.1:0/3579924479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040039a0 0x7f0604003da0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f061c071f10 tx=0x7f06040057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7fc35f5fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 0x7fc358148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f1b0d59a640 1 -- 172.21.15.1:0/2647110451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04031780 con 0x7f1b101622d0 2024-03-24T10:34:01.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7fc35f5fd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 0x7fc35806cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f1b0d59a640 1 -- 172.21.15.1:0/2647110451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b040318e0 con 0x7f1b101622d0 2024-03-24T10:34:01.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f1b0d59a640 1 -- 172.21.15.1:0/2647110451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04031c70 con 0x7f1b101622d0 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7fc35f5fd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3581efc50 con 0x7fc35806d8f0 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7fc35cb98640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 0x7fc358148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35cb98640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 0x7fc358148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48062/0 (socket says 172.21.15.1:48062) 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35cb98640 1 -- 172.21.15.1:0/693576381 learned_addr learned my addr 172.21.15.1:0/693576381 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc35814b9b0 con 0x7fc358148360 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc3581efec0 con 0x7fc358071a50 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35d399640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 0x7fc35814b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808032640 con 0x7f780c004380 2024-03-24T10:34:01.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78080327a0 con 0x7f780c004380 2024-03-24T10:34:01.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f781b7fe640 1 -- 172.21.15.1:0/3104916770 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808032b70 con 0x7f780c004380 2024-03-24T10:34:01.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 0x7fc35806cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.044+0000 7f7b4c8de640 1 -- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 msgr2=0x7f7b400d2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7ffb4c9dc640 1 -- 172.21.15.1:0/2013779680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4402d040 con 0x7ffb4815df60 2024-03-24T10:34:01.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4c8de640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b400d2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7ffb4c9dc640 1 -- 172.21.15.1:0/2013779680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb44031810 con 0x7ffb4815df60 2024-03-24T10:34:01.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4c8de640 1 -- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 msgr2=0x7f7b4004a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7ffb4c9dc640 1 -- 172.21.15.1:0/2013779680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44031ba0 con 0x7ffb4815df60 2024-03-24T10:34:01.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4c8de640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 0x7f7b4004a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4c8de640 1 -- 172.21.15.1:0/754545409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b380094d0 con 0x7f7b4000b0a0 2024-03-24T10:34:01.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f06209c0640 1 -- 172.21.15.1:0/3579924479 reap_dead start 2024-03-24T10:34:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fade1409640 1 Processor -- start 2024-03-24T10:34:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 -- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 msgr2=0x7fc358148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4d8e0640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 0x7f7b4004a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4d0df640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b400d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 shutdown_connections 2024-03-24T10:34:01.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 >> 172.21.15.1:0/3579924479 conn(0x7f061c0d1b30 msgr2=0x7f061c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 0x7fc358148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 -- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 msgr2=0x7fc35814b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 0x7fc35814b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc348009400 con 0x7fc358148360 2024-03-24T10:34:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fade1409640 1 -- start start 2024-03-24T10:34:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 shutdown_connections 2024-03-24T10:34:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35d399640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 0x7fc35814b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7fc35db9a640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 0x7fc35806cb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc35400c3c0 tx=0x7fc35400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f06211c1640 1 -- 172.21.15.1:0/3579924479 wait complete. 2024-03-24T10:34:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f1b1493e640 1 -- 172.21.15.1:0/2647110451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b101622d0 msgr2=0x7f1b100cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/2013779680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 msgr2=0x7ffb480d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb480d0d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffb440097a0 tx=0x7ffb4402b8b0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 msgr2=0x7f780c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 0x7f780c004860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f780800ba30 tx=0x7f780c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 shutdown_connections 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 --2- 172.21.15.1:0/3104916770 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f780c004380 0x7f780c004860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 >> 172.21.15.1:0/3104916770 conn(0x7f7814009fb0 msgr2=0x7f781400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f1b1493e640 1 --2- 172.21.15.1:0/2647110451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b101622d0 0x7f1b100cf840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b040097a0 tx=0x7f1b0402b820 comp rx=0 tx=0).stop 2024-03-24T10:34:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/2013779680 shutdown_connections 2024-03-24T10:34:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb4815e340 0x7ffb480d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/2013779680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb480d0d60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/2013779680 >> 172.21.15.1:0/2013779680 conn(0x7ffb4809d0c0 msgr2=0x7ffb480d3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7ff2f8c64640 1 Processor -- start 2024-03-24T10:34:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7ff2f8c64640 1 -- start start 2024-03-24T10:34:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354005ce0 con 0x7fc358148360 2024-03-24T10:34:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/2013779680 shutdown_connections 2024-03-24T10:34:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/2013779680 wait complete. 2024-03-24T10:34:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc354005e40 con 0x7fc358148360 2024-03-24T10:34:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc350009a70 con 0x7fc358148360 2024-03-24T10:34:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7fade1409640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.045+0000 7f7b4c8de640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7b4807aa50 tx=0x7f7b480695d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3581495d0 con 0x7fc358148360 2024-03-24T10:34:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b480734a0 con 0x7f7b4000b0a0 2024-03-24T10:34:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ff2f8c64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4072900 0x7ff2f4146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3540072d0 con 0x7fc358148360 2024-03-24T10:34:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7ff2f8c64640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 0x7ff2f4143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 -- 172.21.15.1:0/2647110451 shutdown_connections 2024-03-24T10:34:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 --2- 172.21.15.1:0/2647110451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b1006b0a0 0x7f1b1006b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 --2- 172.21.15.1:0/2647110451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b101622d0 0x7f1b100cf840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 -- 172.21.15.1:0/2647110451 >> 172.21.15.1:0/2647110451 conn(0x7f1b100d0b70 msgr2=0x7f1b100d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f8c64640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 0x7ff2f415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 -- 172.21.15.1:0/2647110451 shutdown_connections 2024-03-24T10:34:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f8c64640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f4143b40 con 0x7ff2f4072900 2024-03-24T10:34:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f1b1493e640 1 -- 172.21.15.1:0/2647110451 wait complete. 2024-03-24T10:34:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f06211c1640 1 Processor -- start 2024-03-24T10:34:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f8c64640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f4139880 con 0x7ff2f414b0d0 2024-03-24T10:34:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f8c64640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f4143d70 con 0x7ff2f40b3580 2024-03-24T10:34:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f1d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 0x7ff2f4143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f1d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 0x7ff2f4143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48066/0 (socket says 172.21.15.1:48066) 2024-03-24T10:34:01.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f1d9b640 1 -- 172.21.15.1:0/4162450233 learned_addr learned my addr 172.21.15.1:0/4162450233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7ffb4e9e0640 1 Processor -- start 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 0x7ff2f415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 msgr2=0x7ff2f4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 0x7ff2f4143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4072900 msgr2=0x7ff2f4146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4072900 0x7ff2f4146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7ff2f2d9d640 1 -- 172.21.15.1:0/4162450233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec009b30 con 0x7ff2f414b0d0 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7ff2f2d9d640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 0x7ff2f415ba70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406c7c0 tx=0x7ff2ec004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 shutdown_connections 2024-03-24T10:34:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7821c1f640 1 -- 172.21.15.1:0/3104916770 wait complete. 2024-03-24T10:34:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7ffb4e9e0640 1 -- start start 2024-03-24T10:34:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7f7821c1f640 1 Processor -- start 2024-03-24T10:34:01.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0078b0 con 0x7ff2f414b0d0 2024-03-24T10:34:01.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3540074f0 con 0x7fc358148360 2024-03-24T10:34:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fade1409640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faddc0cd340 0x7faddc161e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fade1409640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faddc14bf20 0x7faddc0c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fade1409640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1623a0 con 0x7faddc06d060 2024-03-24T10:34:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fade1409640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1f9600 con 0x7faddc14bf20 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7fade1409640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faddc1f89f0 con 0x7faddc0cd340 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7faddaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7faddaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33604/0 (socket says 172.21.15.1:33604) 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7faddaffd640 1 -- 172.21.15.1:0/2439907332 learned_addr learned my addr 172.21.15.1:0/2439907332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7fadda7fc640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faddc0cd340 0x7faddc161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.046+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b48070040 con 0x7f7b4000b0a0 2024-03-24T10:34:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b480734a0 con 0x7f7b4000b0a0 2024-03-24T10:34:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b40034de0 con 0x7f7b4000b0a0 2024-03-24T10:34:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.047+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4003a640 con 0x7f7b4000b0a0 2024-03-24T10:34:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7f7b2b7fe640 1 -- 172.21.15.1:0/754545409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b24005290 con 0x7f7b4000b0a0 2024-03-24T10:34:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 0x7ffb48071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc338005290 con 0x7fc358148360 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb481f8250 0x7ffb480ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4806fbe0 con 0x7ffb4815df60 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4806f3f0 con 0x7ffb481f8250 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4e9e0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb4815cbb0 con 0x7ffb480d12a0 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4d1dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4d1dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33620/0 (socket says 172.21.15.1:33620) 2024-03-24T10:34:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ffb4d1dd640 1 -- 172.21.15.1:0/3179266744 learned_addr learned my addr 172.21.15.1:0/3179266744 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.048+0000 7f7821c1f640 1 -- start start 2024-03-24T10:34:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7faddb7fe640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faddc14bf20 0x7faddc0c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f4138f50 con 0x7ff2f414b0d0 2024-03-24T10:34:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f4139510 con 0x7ff2f414b0d0 2024-03-24T10:34:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.049+0000 7f06211c1640 1 -- start start 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 0x7f061c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 0x7f061c15b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0cbab0 con 0x7f061c06ddb0 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f061c15e220 con 0x7f061c0ca990 2024-03-24T10:34:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06211c1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f061c0cae30 con 0x7f061c15de40 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7fc34dffb640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 0x7fc33c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc354089600 con 0x7fc358148360 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7faddaffd640 1 -- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faddc0cd340 msgr2=0x7faddc161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7faddaffd640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faddc0cd340 0x7faddc161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7faddaffd640 1 -- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faddc14bf20 msgr2=0x7faddc0c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7faddaffd640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faddc14bf20 0x7faddc0c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7faddaffd640 1 -- 172.21.15.1:0/2439907332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd4009a70 con 0x7faddc06d060 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2ec007a10 con 0x7ff2f414b0d0 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f1b1493e640 1 Processor -- start 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f1b1493e640 1 -- start start 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 0x7ffb48071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 -- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 msgr2=0x7ffb48071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 0x7ffb48071ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 -- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb481f8250 msgr2=0x7ffb480ccff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb481f8250 0x7ffb480ccff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 -- 172.21.15.1:0/3179266744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb44009400 con 0x7ffb4815df60 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 0x7ffb48071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ffb4d1dd640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000efc0 tx=0x7ffb4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7fc35d399640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 0x7fc33c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec007dc0 con 0x7ff2f414b0d0 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f7821c1f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7814036020 0x7f7814185390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f7821c1f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f7821c1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7814036400 0x7f78140ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f7821c1f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78141862c0 con 0x7f7814036020 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f7821c1f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7814185d00 con 0x7f7814036400 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f7821c1f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7814185fe0 con 0x7f7814178e40 2024-03-24T10:34:01.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7faddaffd640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00c490 tx=0x7fadcc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f781bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7fc35d399640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 0x7fc33c06dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc350036000 tx=0x7fc350030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06209c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06209c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33626/0 (socket says 172.21.15.1:33626) 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.050+0000 7f06209c0640 1 -- 172.21.15.1:0/1481678371 learned_addr learned my addr 172.21.15.1:0/1481678371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f061bfff640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 0x7f061c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f061b7fe640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 0x7f061c15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f06209c0640 1 -- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 msgr2=0x7f061c06d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f06209c0640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 0x7f061c06d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f06209c0640 1 -- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 msgr2=0x7f061c15b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f06209c0640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 0x7f061c15b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f06209c0640 1 -- 172.21.15.1:0/1481678371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0cb110 con 0x7f061c06ddb0 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f061b7fe640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 0x7f061c15b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f781bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48096/0 (socket says 172.21.15.1:48096) 2024-03-24T10:34:01.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f781bfff640 1 -- 172.21.15.1:0/2400448451 learned_addr learned my addr 172.21.15.1:0/2400448451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f061bfff640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 0x7f061c06d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b1493e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b1493e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 0x7f1b100b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b1493e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b101622d0 0x7f1b10164750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b0e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f782141e640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7814036400 0x7f78140ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f7820c1d640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7814036020 0x7f7814185390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b0e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49802/0 (socket says 172.21.15.1:49802) 2024-03-24T10:34:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b0e59c640 1 -- 172.21.15.1:0/4036442240 learned_addr learned my addr 172.21.15.1:0/4036442240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.051+0000 7f1b0dd9b640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 0x7f1b100b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7b48073600 con 0x7f7b4000b0a0 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 -- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7814036400 msgr2=0x7f78140ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7814036400 0x7f78140ea0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 -- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7814036020 msgr2=0x7f7814185390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7814036020 0x7f7814185390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 -- 172.21.15.1:0/2400448451 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781404d1c0 con 0x7f7814178e40 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0ed9d640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b101622d0 0x7f1b10164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1014c5b0 con 0x7f1b10072080 2024-03-24T10:34:01.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f06209c0640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f061c072160 tx=0x7f061400ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10158050 con 0x7f1b1006b0a0 2024-03-24T10:34:01.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d8005290 con 0x7ff2f414b0d0 2024-03-24T10:34:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1b100b6680 con 0x7f1b101622d0 2024-03-24T10:34:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f781bfff640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f780800bb50 tx=0x7f7808004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780803b9b0 con 0x7f7814178e40 2024-03-24T10:34:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780803bb10 con 0x7f7814178e40 2024-03-24T10:34:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780803bea0 con 0x7f7814178e40 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781404d4a0 con 0x7f7814178e40 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0dd9b640 1 -- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b101622d0 msgr2=0x7f1b10164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0dd9b640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b101622d0 0x7f1b10164750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0dd9b640 1 -- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 msgr2=0x7f1b1014c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0dd9b640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0dd9b640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b04009400 con 0x7f1b10072080 2024-03-24T10:34:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7814185ab0 con 0x7f7814178e40 2024-03-24T10:34:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f7b3dffb640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 0x7f7b2c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.052+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7b48104d60 con 0x7f7b4000b0a0 2024-03-24T10:34:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.053+0000 7f7b4d0df640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 0x7f7b2c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.053+0000 7f1b0dd9b640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 0x7f1b100b6140 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000d600 tx=0x7f1b0000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.053+0000 7f7b4d0df640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 0x7f7b2c06dd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b44000f60 tx=0x7f7b44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.053+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781404d660 con 0x7f7814178e40 2024-03-24T10:34:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.054+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f780804d3f0 con 0x7f7814178e40 2024-03-24T10:34:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.054+0000 7f78197fa640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 0x7f780006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.054+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f78080ac5b0 con 0x7f7814178e40 2024-03-24T10:34:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.054+0000 7f7820c1d640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 0x7f780006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.055+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc354054060 con 0x7fc358148360 2024-03-24T10:34:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.055+0000 7f7820c1d640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 0x7f780006dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7814039fb0 tx=0x7f7810009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.056+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff2ec03e450 con 0x7ff2f414b0d0 2024-03-24T10:34:01.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ff2e2ffd640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 0x7ff2d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40009b80 con 0x7ffb4815df60 2024-03-24T10:34:01.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff2ec0acec0 con 0x7ff2f414b0d0 2024-03-24T10:34:01.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ff2f259c640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 0x7ff2d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb480c8f90 con 0x7ffb4815df60 2024-03-24T10:34:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061400aa20 con 0x7f061c06ddb0 2024-03-24T10:34:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7b480cf770 con 0x7f7b4000b0a0 2024-03-24T10:34:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061400ab80 con 0x7f061c06ddb0 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ff2f259c640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 0x7ff2d406dd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40cf8f0 tx=0x7ff2e8007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00014620 con 0x7f1b10072080 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b00014780 con 0x7f1b10072080 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb4806f900 con 0x7ffb4815df60 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4000f040 con 0x7ffb4815df60 2024-03-24T10:34:01.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40014680 con 0x7ffb4815df60 2024-03-24T10:34:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0052d0 con 0x7faddc06d060 2024-03-24T10:34:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc1f8cd0 con 0x7faddc06d060 2024-03-24T10:34:01.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.057+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc1f9290 con 0x7faddc06d060 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c0cb3f0 con 0x7f061c06ddb0 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614007030 con 0x7f061c06ddb0 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c06c1b0 con 0x7f061c06ddb0 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00014b10 con 0x7f1b10072080 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101584e0 con 0x7f1b10072080 2024-03-24T10:34:01.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b10157720 con 0x7f1b10072080 2024-03-24T10:34:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30005290 con 0x7ffb4815df60 2024-03-24T10:34:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadcc005430 con 0x7faddc06d060 2024-03-24T10:34:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.058+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0128f0 con 0x7faddc06d060 2024-03-24T10:34:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.059+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c04f470 con 0x7f061c06ddb0 2024-03-24T10:34:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.060+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b1004f4e0 con 0x7f1b10072080 2024-03-24T10:34:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.060+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb400147e0 con 0x7ffb4815df60 2024-03-24T10:34:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.060+0000 7ffb3e7fc640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 0x7ffb2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.060+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffb4008ad80 con 0x7ffb4815df60 2024-03-24T10:34:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.061+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadbc005290 con 0x7faddc06d060 2024-03-24T10:34:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.061+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 0x7ffb2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.061+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b00014c70 con 0x7f1b10072080 2024-03-24T10:34:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7ffb4d9de640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 0x7ffb2c06dcd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb44036000 tx=0x7ffb4403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0614007190 con 0x7f061c06ddb0 2024-03-24T10:34:01.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f1afeffd640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 0x7f1af006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fadcc012a50 con 0x7faddc06d060 2024-03-24T10:34:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 0x7f1af006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1b0008acf0 con 0x7f1b10072080 2024-03-24T10:34:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f0618ff9640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 0x7f060806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7fadc37fe640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 0x7fadc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0614089c40 con 0x7f061c06ddb0 2024-03-24T10:34:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7f061bfff640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 0x7f060806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.063+0000 7f1b0e59c640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 0x7f1af006dd00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1b0402b860 tx=0x7f1b0403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.062+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fadcc089d40 con 0x7faddc06d060 2024-03-24T10:34:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.063+0000 7fadda7fc640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 0x7fadc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.063+0000 7f061bfff640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 0x7f060806dcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0610036000 tx=0x7f0610030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.063+0000 7fadda7fc640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 0x7fadc806dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadd4036000 tx=0x7fadd4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:01.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.064+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2ec077830 con 0x7ff2f414b0d0 2024-03-24T10:34:01.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.064+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f780807aa50 con 0x7f7814178e40 2024-03-24T10:34:01.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.066+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f06140547d0 con 0x7f061c06ddb0 2024-03-24T10:34:01.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.075+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb40055700 con 0x7ffb4815df60 2024-03-24T10:34:01.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.083+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b000557a0 con 0x7f1b10072080 2024-03-24T10:34:01.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.092+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fadcc0546c0 con 0x7faddc06d060 2024-03-24T10:34:01.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.462+0000 7f7b2b7fe640 1 -- 172.21.15.1:0/754545409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7b24005810 con 0x7f7b4000b0a0 2024-03-24T10:34:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.463+0000 7f7b3dffb640 1 -- 172.21.15.1:0/754545409 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7b48098090 con 0x7f7b4000b0a0 2024-03-24T10:34:01.470 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.466+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 msgr2=0x7f7b2c06dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 0x7f7b2c06dd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b44000f60 tx=0x7f7b44009040 comp rx=0 tx=0).stop 2024-03-24T10:34:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 msgr2=0x7f7b4003a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7b4807aa50 tx=0x7f7b480695d0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 shutdown_connections 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7b2c06b8d0 0x7f7b2c06dd90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400e7750 0x7f7b4004a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.467+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b4000b0a0 0x7f7b4003a100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.468+0000 7f7b4f343640 1 --2- 172.21.15.1:0/754545409 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7b40007330 0x7f7b400d2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.468+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 >> 172.21.15.1:0/754545409 conn(0x7f7b4000a3f0 msgr2=0x7f7b4004f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.468+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 shutdown_connections 2024-03-24T10:34:01.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.468+0000 7f7b4f343640 1 -- 172.21.15.1:0/754545409 wait complete. 2024-03-24T10:34:01.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.482+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f061c0c51e0 con 0x7f061c06ddb0 2024-03-24T10:34:01.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.483+0000 7f0618ff9640 1 -- 172.21.15.1:0/1481678371 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0614018090 con 0x7f061c06ddb0 2024-03-24T10:34:01.484 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-24T10:34:01.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-24T10:34:01.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.485+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc338005540 con 0x7fc358148360 2024-03-24T10:34:01.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7fc34dffb640 1 -- 172.21.15.1:0/693576381 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc354017090 con 0x7fc358148360 2024-03-24T10:34:01.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f78140052c0 con 0x7f7814178e40 2024-03-24T10:34:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 msgr2=0x7f060806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 0x7f060806dcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0610036000 tx=0x7f0610030000 comp rx=0 tx=0).stop 2024-03-24T10:34:01.488 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 msgr2=0x7f061c15b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f061c072160 tx=0x7f061400ce40 comp rx=0 tx=0).stop 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 shutdown_connections 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060806b810 0x7f060806dcd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06ddb0 0x7f061c15b950 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0ca990 0x7f061c15b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 --2- 172.21.15.1:0/1481678371 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c15de40 0x7f061c06d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 >> 172.21.15.1:0/1481678371 conn(0x7f061c0d1b30 msgr2=0x7f061c0b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.486+0000 7f78197fa640 1 -- 172.21.15.1:0/2400448451 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7808036020 con 0x7f7814178e40 2024-03-24T10:34:01.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.487+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 shutdown_connections 2024-03-24T10:34:01.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.487+0000 7f06211c1640 1 -- 172.21.15.1:0/1481678371 wait complete. 2024-03-24T10:34:01.492 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.488+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 msgr2=0x7fc33c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.488+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 0x7fc33c06dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc350036000 tx=0x7fc350030000 comp rx=0 tx=0).stop 2024-03-24T10:34:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.488+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 msgr2=0x7fc35806cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.488+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 0x7fc35806cb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc35400c3c0 tx=0x7fc35400c880 comp rx=0 tx=0).stop 2024-03-24T10:34:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 shutdown_connections 2024-03-24T10:34:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc33c06b8e0 0x7fc33c06dda0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc358148360 0x7fc35806cb20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc358071a50 0x7fc358148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 --2- 172.21.15.1:0/693576381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc35806d8f0 0x7fc35814b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 >> 172.21.15.1:0/693576381 conn(0x7fc35806b0a0 msgr2=0x7fc35806e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff2d8005810 con 0x7ff2f414b0d0 2024-03-24T10:34:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 shutdown_connections 2024-03-24T10:34:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7fc35f5fd640 1 -- 172.21.15.1:0/693576381 wait complete. 2024-03-24T10:34:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.489+0000 7ff2e2ffd640 1 -- 172.21.15.1:0/4162450233 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff2ec043070 con 0x7ff2f414b0d0 2024-03-24T10:34:01.496 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.490+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 msgr2=0x7f780006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.490+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 0x7f780006dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7814039fb0 tx=0x7f7810009040 comp rx=0 tx=0).stop 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.490+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 msgr2=0x7f78140e9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.490+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f780800bb50 tx=0x7f7808004730 comp rx=0 tx=0).stop 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 shutdown_connections 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f780006b840 0x7f780006dd00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7814036400 0x7f78140ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7814178e40 0x7f78140e9b80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 --2- 172.21.15.1:0/2400448451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7814036020 0x7f7814185390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 >> 172.21.15.1:0/2400448451 conn(0x7f7814009fb0 msgr2=0x7f7814048590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 shutdown_connections 2024-03-24T10:34:01.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.491+0000 7f7821c1f640 1 -- 172.21.15.1:0/2400448451 wait complete. 2024-03-24T10:34:01.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 msgr2=0x7ff2d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 0x7ff2d406dd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40cf8f0 tx=0x7ff2e8007970 comp rx=0 tx=0).stop 2024-03-24T10:34:01.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 msgr2=0x7ff2f415ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 0x7ff2f415ba70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406c7c0 tx=0x7ff2ec004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 shutdown_connections 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff2d406b860 0x7ff2d406dd20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f414b0d0 0x7ff2f415ba70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff2f40b3580 0x7ff2f4143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 --2- 172.21.15.1:0/4162450233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4072900 0x7ff2f4146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 >> 172.21.15.1:0/4162450233 conn(0x7ff2f406b0a0 msgr2=0x7ff2f415fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 shutdown_connections 2024-03-24T10:34:01.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.492+0000 7ff2f8c64640 1 -- 172.21.15.1:0/4162450233 wait complete. 2024-03-24T10:34:01.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.495+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ffb30005540 con 0x7ffb4815df60 2024-03-24T10:34:01.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.496+0000 7ffb3e7fc640 1 -- 172.21.15.1:0/3179266744 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffb40058dc0 con 0x7ffb4815df60 2024-03-24T10:34:01.503 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.498+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 msgr2=0x7ffb2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.498+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 0x7ffb2c06dcd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb44036000 tx=0x7ffb4403c040 comp rx=0 tx=0).stop 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.498+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 msgr2=0x7ffb4806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.498+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000efc0 tx=0x7ffb4000c490 comp rx=0 tx=0).stop 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 shutdown_connections 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb2c06b810 0x7ffb2c06dcd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb481f8250 0x7ffb480ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4815df60 0x7ffb4806eeb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 --2- 172.21.15.1:0/3179266744 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb480d12a0 0x7ffb48071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 >> 172.21.15.1:0/3179266744 conn(0x7ffb4809d0c0 msgr2=0x7ffb480b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 shutdown_connections 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.499+0000 7ffb4e9e0640 1 -- 172.21.15.1:0/3179266744 wait complete. 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.503+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1b100b68b0 con 0x7f1b10072080 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.504+0000 7f1afeffd640 1 -- 172.21.15.1:0/4036442240 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1b00058e60 con 0x7f1b10072080 2024-03-24T10:34:01.506 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:01.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.507+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 msgr2=0x7f1af006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.507+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 0x7f1af006dd00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1b0402b860 tx=0x7f1b0403d040 comp rx=0 tx=0).stop 2024-03-24T10:34:01.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.507+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 msgr2=0x7f1b100b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.507+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 0x7f1b100b6140 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000d600 tx=0x7f1b0000dac0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2024-03-24T10:34:01.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 shutdown_connections 2024-03-24T10:34:01.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1af006b840 0x7f1af006dd00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1b101622d0 0x7f1b10164750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b10072080 0x7f1b100b6140 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 --2- 172.21.15.1:0/4036442240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1006b0a0 0x7f1b1014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 >> 172.21.15.1:0/4036442240 conn(0x7f1b100d0b70 msgr2=0x7f1b10160020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 shutdown_connections 2024-03-24T10:34:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.509+0000 7f1b1493e640 1 -- 172.21.15.1:0/4036442240 wait complete. 2024-03-24T10:34:01.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-24T10:34:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.510+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fadbc005540 con 0x7faddc06d060 2024-03-24T10:34:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.510+0000 7fadc37fe640 1 -- 172.21.15.1:0/2439907332 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fadcc057d80 con 0x7faddc06d060 2024-03-24T10:34:01.514 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:01.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-24T10:34:01.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 msgr2=0x7fadc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 0x7fadc806dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadd4036000 tx=0x7fadd4030000 comp rx=0 tx=0).stop 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 msgr2=0x7faddc1f8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00c490 tx=0x7fadcc00c950 comp rx=0 tx=0).stop 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 shutdown_connections 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fadc806b810 0x7fadc806dcd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faddc14bf20 0x7faddc0c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faddc0cd340 0x7faddc161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 --2- 172.21.15.1:0/2439907332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faddc06d060 0x7faddc1f8410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 >> 172.21.15.1:0/2439907332 conn(0x7faddc06b0a0 msgr2=0x7faddc0ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 shutdown_connections 2024-03-24T10:34:01.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-24T10:34:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:01.514+0000 7fade1409640 1 -- 172.21.15.1:0/2439907332 wait complete. 2024-03-24T10:34:01.528 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-24T10:34:01.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2024-03-24T10:34:02.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T10:34:02.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T10:34:02.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T10:34:02.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T10:34:02.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T10:34:02.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T10:34:02.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T10:34:02.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T10:34:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.651+0000 7fb9326b7640 1 Processor -- start 2024-03-24T10:34:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.651+0000 7fb9326b7640 1 -- start start 2024-03-24T10:34:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.652+0000 7fb9326b7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb92c08b2f0 0x7fb92c08b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.652+0000 7fb9326b7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c16aea0 0x7fb92c0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.652+0000 7fb9326b7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c072cc0 con 0x7fb92c070b00 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.652+0000 7fb9326b7640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c08bc10 con 0x7fb92c08b2f0 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.652+0000 7fb9326b7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c06c920 con 0x7fb92c16aea0 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.653+0000 7fb931eb6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb92c070b00 0x7fb92c06d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41602/0 (socket says 172.21.15.1:41602) 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.653+0000 7fb931eb6640 1 -- 172.21.15.1:0/3955804632 learned_addr learned my addr 172.21.15.1:0/3955804632 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.653+0000 7fb930eb4640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c16aea0 0x7fb92c0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.653+0000 7fb9316b5640 1 --2- 172.21.15.1:0/3955804632 >> v2:172.21.15.3:3300/0 conn(0x7fb92c08b2f0 0x7fb92c08b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014734517 0 0) 0x7fb92c072cc0 con 0x7fb92c070b00 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb918007930 con 0x7fb92c070b00 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1797662833 0 0) 0x7fb918007930 con 0x7fb92c070b00 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c16aea0 msgr2=0x7fb92c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c16aea0 0x7fb92c0cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 >> v2:172.21.15.3:3300/0 conn(0x7fb92c08b2f0 msgr2=0x7fb92c08b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 --2- 172.21.15.1:0/3955804632 >> v2:172.21.15.3:3300/0 conn(0x7fb92c08b2f0 0x7fb92c08b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c06cc00 con 0x7fb92c070b00 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fb9240019f0 con 0x7fb92c070b00 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 >> v1:172.21.15.1:6789/0 conn(0x7fb92c070b00 legacy=0x7fb92c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918007d80 0x7fb9180081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb918008bd0 0x7fb918009050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb918007440 0x7fb918009590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c08bc10 con 0x7fb918008bd0 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c06c920 con 0x7fb918007440 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.654+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb91800bfe0 con 0x7fb918007d80 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.655+0000 7fb930eb4640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb918008bd0 0x7fb918009050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.655+0000 7fb931eb6640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb918007440 0x7fb918009590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.656+0000 7fb930eb4640 1 -- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918007d80 msgr2=0x7fb9180081a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.656+0000 7fb930eb4640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918007d80 0x7fb9180081a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.656+0000 7fb930eb4640 1 -- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb918007440 msgr2=0x7fb918009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.656+0000 7fb930eb4640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb918007440 0x7fb918009590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.656+0000 7fb930eb4640 1 -- 172.21.15.1:0/3955804632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c06cc00 con 0x7fb918008bd0 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.657+0000 7fb930eb4640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb918008bd0 0x7fb918009050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0cc280 tx=0x7fb918009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4c79bbee682f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.658+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928030660 con 0x7fb918008bd0 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.658+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9280307c0 con 0x7fb918008bd0 2024-03-24T10:34:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.658+0000 7fb923fff640 1 -- 172.21.15.1:0/3955804632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928030b70 con 0x7fb918008bd0 2024-03-24T10:34:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.659+0000 7fb9326b7640 1 -- 172.21.15.1:0/3955804632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb918008bd0 msgr2=0x7fb918009050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.659+0000 7fb9326b7640 1 --2- 172.21.15.1:0/3955804632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb918008bd0 0x7fb918009050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0cc280 tx=0x7fb918009c00 comp rx=0 tx=0).stop 2024-03-24T10:34:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.660+0000 7fb931eb6640 1 -- 172.21.15.1:0/3955804632 reap_dead start 2024-03-24T10:34:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.660+0000 7fb9326b7640 1 -- 172.21.15.1:0/3955804632 shutdown_connections 2024-03-24T10:34:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.660+0000 7fb9326b7640 1 -- 172.21.15.1:0/3955804632 >> 172.21.15.1:0/3955804632 conn(0x7fb92c0b16a0 msgr2=0x7fb92c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.660+0000 7fb9326b7640 1 -- 172.21.15.1:0/3955804632 shutdown_connections 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.660+0000 7fb9326b7640 1 -- 172.21.15.1:0/3955804632 wait complete. 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.662+0000 7fb9326b7640 1 Processor -- start 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.662+0000 7fb9326b7640 1 -- start start 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c0ab080 0x7fb92c16cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 0x7fb92c0cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c200de0 con 0x7fb92c16d260 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c164500 con 0x7fb92c0ab080 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9326b7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c1649a0 con 0x7fb918008bd0 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.663+0000 7fb9316b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48142/0 (socket says 172.21.15.1:48142) 2024-03-24T10:34:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 -- 172.21.15.1:0/2130812003 learned_addr learned my addr 172.21.15.1:0/2130812003 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb931eb6640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 0x7fb92c0cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 -- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c0ab080 msgr2=0x7fb92c16cd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:34:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c0ab080 0x7fb92c16cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 -- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 msgr2=0x7fb92c0cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 0x7fb92c0cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb9316b5640 1 -- 172.21.15.1:0/2130812003 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c0a9fe0 con 0x7fb918008bd0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.664+0000 7fb931eb6640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 0x7fb92c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.665+0000 7fb9316b5640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00d9c0 tx=0x7fb91c00de80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.666+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c007b10 con 0x7fb918008bd0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.666+0000 7fb9326b7640 1 -- 172.21.15.1:0/2130812003 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c0aa2c0 con 0x7fb918008bd0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.666+0000 7fb9326b7640 1 -- 172.21.15.1:0/2130812003 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c0aa880 con 0x7fb918008bd0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.667+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb91c007c70 con 0x7fb918008bd0 2024-03-24T10:34:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.667+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c010470 con 0x7fb918008bd0 2024-03-24T10:34:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.668+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb91c0105d0 con 0x7fb918008bd0 2024-03-24T10:34:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.670+0000 7fb921ffb640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 0x7fb9180807d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.670+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb91c08a480 con 0x7fb918008bd0 2024-03-24T10:34:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.670+0000 7fb930eb4640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 0x7fb9180807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.670+0000 7fb9326b7640 1 -- 172.21.15.1:0/2130812003 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb910005290 con 0x7fb918008bd0 2024-03-24T10:34:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.671+0000 7fb930eb4640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 0x7fb9180807d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0734a0 tx=0x7fb928030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.681+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb91c01f080 con 0x7fb918008bd0 2024-03-24T10:34:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.704+0000 7fe436f75640 1 Processor -- start 2024-03-24T10:34:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.704+0000 7fe436f75640 1 -- start start 2024-03-24T10:34:02.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.706+0000 7fe436f75640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 0x7fe4300cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.706+0000 7fe436f75640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 0x7fe430070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.706+0000 7fe435f73640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 0x7fe4300cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.706+0000 7fe435f73640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 0x7fe4300cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48152/0 (socket says 172.21.15.1:48152) 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe435772640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 0x7fe430070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe435772640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 0x7fe430070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33678/0 (socket says 172.21.15.1:33678) 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe435f73640 1 -- 172.21.15.1:0/1624340406 learned_addr learned my addr 172.21.15.1:0/1624340406 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe430072cc0 con 0x7fe43008b2f0 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe43006c920 con 0x7fe430070b00 2024-03-24T10:34:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.707+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe43006cc00 con 0x7fe43016ac30 2024-03-24T10:34:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952071244 0 0) 0x7fe430072cc0 con 0x7fe43008b2f0 2024-03-24T10:34:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7f24a1531640 1 Processor -- start 2024-03-24T10:34:02.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7f24a1531640 1 -- start start 2024-03-24T10:34:02.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7fd66996c640 1 Processor -- start 2024-03-24T10:34:02.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7fd66996c640 1 -- start start 2024-03-24T10:34:02.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7f98f9d92640 1 Processor -- start 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe41c003520 con 0x7fe43008b2f0 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2395456181 0 0) 0x7fe41c003520 con 0x7fe43008b2f0 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 msgr2=0x7fe4300cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 0x7fe4300cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 msgr2=0x7fe430070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 --2- 172.21.15.1:0/1624340406 >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 0x7fe430070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43006cee0 con 0x7fe43008b2f0 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fe4280019f0 con 0x7fe43008b2f0 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.708+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 >> v1:172.21.15.1:6789/0 conn(0x7fe43008b2f0 legacy=0x7fe43006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe41c003970 0x7fe41c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c0047c0 0x7fe41c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe41c014580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe43006c920 con 0x7fe41c005180 2024-03-24T10:34:02.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe43006cc00 con 0x7fe41c0047c0 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe41c009c00 con 0x7fe41c003970 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe436774640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe41c014580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe435f73640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016ac30 0x7fe4300cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe435772640 1 --2- 172.21.15.1:0/1624340406 >> v2:172.21.15.1:3300/0 conn(0x7fe430070b00 0x7fe430070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe436774640 1 -- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe41c003970 msgr2=0x7fe41c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe435f73640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe41c003970 0x7fe41c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.709+0000 7fe435772640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c0047c0 0x7fe41c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe436774640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe41c003970 0x7fe41c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe436774640 1 -- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c0047c0 msgr2=0x7fe41c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe436774640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c0047c0 0x7fe41c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe436774640 1 -- 172.21.15.1:0/1624340406 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43006cee0 con 0x7fe41c005180 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe435f73640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe41c003970 0x7fe41c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.710+0000 7fe435772640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c0047c0 0x7fe41c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7f43095c8640 1 Processor -- start 2024-03-24T10:34:02.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7f24a1531640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f24a1531640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2494008f40 0x7f249403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48160/0 (socket says 172.21.15.1:48160) 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 -- 172.21.15.1:0/2297200024 learned_addr learned my addr 172.21.15.1:0/2297200024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249b7fe640 1 --2- 172.21.15.1:0/2297200024 >> v2:172.21.15.3:3300/0 conn(0x7f2494008f40 0x7f249403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f43095c8640 1 -- start start 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249400b700 con 0x7f249403adf0 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f249403b280 con 0x7f2494008f40 2024-03-24T10:34:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f249400cb20 con 0x7f24940e0010 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.711+0000 7f98f9d92640 1 -- start start 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd66406ebf0 0x7fd6640b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 0x7fd664140370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 0x7fd664070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd6640714d0 con 0x7fd664143230 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd66406cc50 con 0x7fd66406ebf0 2024-03-24T10:34:02.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fd66996c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd6641408b0 con 0x7fd664070b00 2024-03-24T10:34:02.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f98f9d92640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98f40d2a40 0x7f98f40b4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fef57fff640 1 Processor -- start 2024-03-24T10:34:02.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fe436774640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe41c014580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c740 tx=0x7fe41c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfdfc8d2fa7d82e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43095c8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 0x7f42fc04fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43095c8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42fc054320 con 0x7f42fc04ffc0 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43095c8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42fc053710 con 0x7f42fc006ee0 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43095c8640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc0539f0 con 0x7f42fc0eacf0 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43037fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42fc04ffc0 0x7f42fc037bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41638/0 (socket says 172.21.15.1:41638) 2024-03-24T10:34:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f43037fe640 1 -- 172.21.15.1:0/1144940669 learned_addr learned my addr 172.21.15.1:0/1144940669 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 -- 172.21.15.1:0/2297200024 >> v2:172.21.15.3:3300/0 conn(0x7f2494008f40 msgr2=0x7f249403a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 --2- 172.21.15.1:0/2297200024 >> v2:172.21.15.3:3300/0 conn(0x7f2494008f40 0x7f249403a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 -- 172.21.15.1:0/2297200024 >> v1:172.21.15.3:6789/0 conn(0x7f249403adf0 legacy=0x7f24940e3bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f249bfff640 1 -- 172.21.15.1:0/2297200024 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24940565c0 con 0x7f24940e0010 2024-03-24T10:34:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7fef57fff640 1 -- start start 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd66406ebf0 0x7fd6640b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd66406ebf0 0x7fd6640b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49830/0 (socket says 172.21.15.1:49830) 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 -- 172.21.15.1:0/1590052088 learned_addr learned my addr 172.21.15.1:0/1590052088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd6637fe640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 0x7fd664070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd6627fc640 1 --2- 172.21.15.1:0/1590052088 >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 0x7fd664140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f98f9d92640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98f40cfe50 con 0x7f98f415df60 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f98f9d92640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98f40cedb0 con 0x7f98f415e340 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.712+0000 7f98f9d92640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f98f40d49a0 con 0x7f98f40d2a40 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98f40d2a40 0x7f98f40b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98f40d2a40 0x7f98f40b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33710/0 (socket says 172.21.15.1:33710) 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 -- 172.21.15.1:0/5481087 learned_addr learned my addr 172.21.15.1:0/5481087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f4303fff640 1 --2- 172.21.15.1:0/1144940669 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 0x7f42fc04fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 -- 172.21.15.1:0/5481087 >> v1:172.21.15.3:6789/0 conn(0x7f98f415e340 legacy=0x7f98f40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 -- 172.21.15.1:0/5481087 >> v1:172.21.15.1:6789/0 conn(0x7f98f415df60 legacy=0x7f98f40d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7f98f9591640 1 -- 172.21.15.1:0/5481087 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f40cf090 con 0x7f98f40d2a40 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428033600 con 0x7fe41c005180 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42802f050 con 0x7fe41c005180 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fe434f71640 1 -- 172.21.15.1:0/1624340406 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428033a00 con 0x7fe41c005180 2024-03-24T10:34:02.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f249bfff640 1 --2- 172.21.15.1:0/2297200024 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24900098a0 tx=0x7f249002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f195084f185b6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198208223 0 0) 0x7f42fc053710 con 0x7f42fc006ee0 2024-03-24T10:34:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 -- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 msgr2=0x7fd664070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 0x7fd664070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 -- 172.21.15.1:0/1590052088 >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 msgr2=0x7fd664140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fef57fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef50038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f249affd640 1 -- 172.21.15.1:0/2297200024 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490031870 con 0x7f24940e0010 2024-03-24T10:34:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 --2- 172.21.15.1:0/1590052088 >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 0x7fd664140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd662ffd640 1 -- 172.21.15.1:0/1590052088 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd664160e50 con 0x7fd66406ebf0 2024-03-24T10:34:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.713+0000 7fd6637fe640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 0x7fd664070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f4003520 con 0x7f42fc006ee0 2024-03-24T10:34:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4303fff640 1 -- 172.21.15.1:0/1144940669 >> v1:172.21.15.3:6789/0 conn(0x7f42fc006ee0 legacy=0x7f42fc0382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4303fff640 1 -- 172.21.15.1:0/1144940669 >> v1:172.21.15.1:6789/0 conn(0x7f42fc04ffc0 legacy=0x7f42fc037bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fd6627fc640 1 --2- 172.21.15.1:0/1590052088 >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 0x7fd664140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118625089 0 0) 0x7f42fc054320 con 0x7f42fc04ffc0 2024-03-24T10:34:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f249affd640 1 -- 172.21.15.1:0/2297200024 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24900319d0 con 0x7f24940e0010 2024-03-24T10:34:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f249affd640 1 -- 172.21.15.1:0/2297200024 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490031da0 con 0x7f24940e0010 2024-03-24T10:34:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fef56ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef50038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fef56ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef50038080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48176/0 (socket says 172.21.15.1:48176) 2024-03-24T10:34:02.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fd662ffd640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd66406ebf0 0x7fd6640b3910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd664068d90 tx=0x7fd65402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cc8cd78b331c629 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7fef57fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef50053c70 0x7fef5004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef56ffd640 1 -- 172.21.15.1:0/4163491964 learned_addr learned my addr 172.21.15.1:0/4163491964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fd661ffb640 1 -- 172.21.15.1:0/1590052088 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654030800 con 0x7fd66406ebf0 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fd661ffb640 1 -- 172.21.15.1:0/1590052088 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd654030960 con 0x7fd66406ebf0 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fd661ffb640 1 -- 172.21.15.1:0/1590052088 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654030d30 con 0x7fd66406ebf0 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 msgr2=0x7fe41c014580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fe436f75640 1 --2- 172.21.15.1:0/1624340406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe41c014580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c740 tx=0x7fe41c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fe436774640 1 -- 172.21.15.1:0/1624340406 reap_dead start 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef567fc640 1 --2- 172.21.15.1:0/4163491964 >> v2:172.21.15.3:3300/0 conn(0x7fef50053c70 0x7fef5004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef500544c0 con 0x7fef5000c0a0 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fef500541e0 con 0x7fef50053c70 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fef50005f80 con 0x7fef500df220 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f98f9591640 1 --2- 172.21.15.1:0/5481087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40b4960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009e50 tx=0x7f98ec02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a6e79260c14bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.714+0000 7f4303fff640 1 -- 172.21.15.1:0/1144940669 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc053cd0 con 0x7f42fc0eacf0 2024-03-24T10:34:02.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 msgr2=0x7f249400b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f98f37fe640 1 -- 172.21.15.1:0/5481087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec02cbb0 con 0x7f98f40d2a40 2024-03-24T10:34:02.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 --2- 172.21.15.1:0/2297200024 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24900098a0 tx=0x7f249002b910 comp rx=0 tx=0).stop 2024-03-24T10:34:02.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f98f37fe640 1 -- 172.21.15.1:0/5481087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ec02cd10 con 0x7f98f40d2a40 2024-03-24T10:34:02.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f98f37fe640 1 -- 172.21.15.1:0/5481087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec02cbb0 con 0x7f98f40d2a40 2024-03-24T10:34:02.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43340860 0 0) 0x7fef500544c0 con 0x7fef5000c0a0 2024-03-24T10:34:02.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef40003420 con 0x7fef5000c0a0 2024-03-24T10:34:02.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 shutdown_connections 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 >> 172.21.15.1:0/1624340406 conn(0x7fe4300b16a0 msgr2=0x7fe4300b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f4303fff640 1 --2- 172.21.15.1:0/1144940669 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 0x7f42fc04fa80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097d0 tx=0x7f42f802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f11277452262925c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 shutdown_connections 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 --2- 172.21.15.1:0/2297200024 >> v2:172.21.15.3:3300/0 conn(0x7f2494008f40 0x7f249403a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 --2- 172.21.15.1:0/2297200024 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f24940e0010 0x7f249400b1c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.715+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 >> 172.21.15.1:0/2297200024 conn(0x7f249401ef60 msgr2=0x7f2494055c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 shutdown_connections 2024-03-24T10:34:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.1 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2362727875 0 0) 0x7fef40003420 con 0x7fef5000c0a0 2024-03-24T10:34:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 msgr2=0x7fef50038080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef50038080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 >> v2:172.21.15.3:3300/0 conn(0x7fef50053c70 msgr2=0x7fef5004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 --2- 172.21.15.1:0/4163491964 >> v2:172.21.15.3:3300/0 conn(0x7fef50053c70 0x7fef5004fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 -- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd66406ebf0 msgr2=0x7fd6640b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd66406ebf0 0x7fd6640b3910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd664068d90 tx=0x7fd65402b940 comp rx=0 tx=0).stop 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 -- 172.21.15.1:0/1590052088 shutdown_connections 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd664070b00 0x7fd664070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 --2- 172.21.15.1:0/1590052088 >> v2:172.21.15.1:3300/0 conn(0x7fd664143230 0x7fd664140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 --2- 172.21.15.1:0/1590052088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd66406ebf0 0x7fd6640b3910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 -- 172.21.15.1:0/1590052088 >> 172.21.15.1:0/1590052088 conn(0x7fd66406b0a0 msgr2=0x7fd66406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 -- 172.21.15.1:0/1590052088 shutdown_connections 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fd66996c640 1 -- 172.21.15.1:0/1590052088 wait complete. 2024-03-24T10:34:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f24a1531640 1 -- 172.21.15.1:0/2297200024 wait complete. 2024-03-24T10:34:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f98f9d92640 1 -- 172.21.15.1:0/5481087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 msgr2=0x7f98f40b4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f98f9d92640 1 --2- 172.21.15.1:0/5481087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40b4960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009e50 tx=0x7f98ec02c010 comp rx=0 tx=0).stop 2024-03-24T10:34:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8030800 con 0x7f42fc0eacf0 2024-03-24T10:34:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f8030960 con 0x7f42fc0eacf0 2024-03-24T10:34:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f4302ffd640 1 -- 172.21.15.1:0/1144940669 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8030cf0 con 0x7f42fc0eacf0 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 shutdown_connections 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef50006260 con 0x7fef5000c0a0 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.1 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fef5804f990 con 0x7fef5000c0a0 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 >> v1:172.21.15.3:6789/0 conn(0x7fef5000c0a0 legacy=0x7fef50006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef40004a50 0x7fef40004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef40003660 0x7fef40003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef40004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fef500541e0 con 0x7fef40003660 2024-03-24T10:34:02.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fef50005f80 con 0x7fef40004a50 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fef4000fcf0 con 0x7fef500df220 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef567fc640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef40003660 0x7fef40003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef40004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f98f9d92640 1 -- 172.21.15.1:0/5481087 shutdown_connections 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f98f9d92640 1 --2- 172.21.15.1:0/5481087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40b4960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7f98f9d92640 1 -- 172.21.15.1:0/5481087 >> 172.21.15.1:0/5481087 conn(0x7f98f409d0c0 msgr2=0x7f98f40d3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7f98f9d92640 1 -- 172.21.15.1:0/5481087 shutdown_connections 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef56ffd640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef40004a50 0x7fef40004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 -- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef40004a50 msgr2=0x7fef40004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef40004a50 0x7fef40004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 -- 172.21.15.1:0/4163491964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef40003660 msgr2=0x7fef40003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef40003660 0x7fef40003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.716+0000 7fef577fe640 1 -- 172.21.15.1:0/4163491964 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef40003420 con 0x7fef500df220 2024-03-24T10:34:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7fe436f75640 1 -- 172.21.15.1:0/1624340406 wait complete. 2024-03-24T10:34:02.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7f85a5286640 1 Processor -- start 2024-03-24T10:34:02.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7f85a5286640 1 -- start start 2024-03-24T10:34:02.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7f98f9d92640 1 -- 172.21.15.1:0/5481087 wait complete. 2024-03-24T10:34:02.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.717+0000 7f43095c8640 1 -- 172.21.15.1:0/1144940669 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 msgr2=0x7f42fc04fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fd66996c640 1 Processor -- start 2024-03-24T10:34:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f43095c8640 1 --2- 172.21.15.1:0/1144940669 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 0x7f42fc04fa80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097d0 tx=0x7f42f802b940 comp rx=0 tx=0).stop 2024-03-24T10:34:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fef577fe640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef40004020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef580699a0 tx=0x7fef400046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f12bda37a75b440f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580af440 con 0x7fef500df220 2024-03-24T10:34:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fd66996c640 1 -- start start 2024-03-24T10:34:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef58070040 con 0x7fef500df220 2024-03-24T10:34:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7fef55ffb640 1 -- 172.21.15.1:0/4163491964 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580af7e0 con 0x7fef500df220 2024-03-24T10:34:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a5286640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598037680 0x7f8598057680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a5286640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8598052480 0x7f85980528f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a5286640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8598038200 con 0x7f8598037a60 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a5286640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f8598055eb0 con 0x7f8598052480 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a5286640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f85980558f0 con 0x7f8598037680 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f24a1531640 1 Processor -- start 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f98f9d92640 1 Processor -- start 2024-03-24T10:34:02.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f98f9d92640 1 -- start start 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f43095c8640 1 -- 172.21.15.1:0/1144940669 shutdown_connections 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f43095c8640 1 --2- 172.21.15.1:0/1144940669 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc0eacf0 0x7f42fc04fa80 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097d0 tx=0x7f42f802b940 comp rx=0 tx=0).stop 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f43095c8640 1 -- 172.21.15.1:0/1144940669 >> 172.21.15.1:0/1144940669 conn(0x7f42fc009fa0 msgr2=0x7f42fc00a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fe436f75640 1 Processor -- start 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fe436f75640 1 -- start start 2024-03-24T10:34:02.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a4a85640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8598052480 0x7f85980528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a4a85640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8598052480 0x7f85980528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49850/0 (socket says 172.21.15.1:49850) 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f85a4a85640 1 -- 172.21.15.1:0/960131947 learned_addr learned my addr 172.21.15.1:0/960131947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.718+0000 7f859ffff640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598037680 0x7f8598057680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7f24a1531640 1 -- start start 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7f85a4a85640 1 -- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598037680 msgr2=0x7f8598057680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:02.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7f85a4a85640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598037680 0x7f8598057680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7f85a4a85640 1 -- 172.21.15.1:0/960131947 >> v1:172.21.15.3:6789/0 conn(0x7f8598037a60 legacy=0x7f85980366a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:34:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7f85a4a85640 1 -- 172.21.15.1:0/960131947 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8598055bd0 con 0x7f8598052480 2024-03-24T10:34:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 msgr2=0x7fef40004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef57fff640 1 --2- 172.21.15.1:0/4163491964 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef40004020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef580699a0 tx=0x7fef400046e0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef577fe640 1 -- 172.21.15.1:0/4163491964 reap_dead start 2024-03-24T10:34:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fd66996c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 shutdown_connections 2024-03-24T10:34:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 >> 172.21.15.1:0/4163491964 conn(0x7fef50009fb0 msgr2=0x7fef5000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fd66996c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd664070b00 0x7fd664135230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.719+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 shutdown_connections 2024-03-24T10:34:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fef57fff640 1 -- 172.21.15.1:0/4163491964 wait complete. 2024-03-24T10:34:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd662ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd662ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48214/0 (socket says 172.21.15.1:48214) 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 0x7f98f406f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98f40c9f80 con 0x7f98f40d2a40 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe436f75640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe43016bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f98f40c76a0 con 0x7f98f415df60 2024-03-24T10:34:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9d92640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f98f40c7940 con 0x7f98f41f8720 2024-03-24T10:34:02.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f24a1531640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2494008f40 0x7f24940cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f85a4a85640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598052480 0x7f85980528f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f858c009870 tx=0x7f858c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f181d0963d4718de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe435f73640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe43016bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe435f73640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe43016bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33726/0 (socket says 172.21.15.1:33726) 2024-03-24T10:34:02.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe435f73640 1 -- 172.21.15.1:0/1209516742 learned_addr learned my addr 172.21.15.1:0/1209516742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd662ffd640 1 -- 172.21.15.1:0/3707616343 learned_addr learned my addr 172.21.15.1:0/3707616343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd6627fc640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd664070b00 0x7fd664135230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd664143230 0x7fd6641505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9591640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 0x7f98f406f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f8d90640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49864/0 (socket says 172.21.15.1:49864) 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f9591640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 0x7f98f406f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48216/0 (socket says 172.21.15.1:48216) 2024-03-24T10:34:02.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f8d90640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33714/0 (socket says 172.21.15.1:33714) 2024-03-24T10:34:02.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 0x7fe43008ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f98f3fff640 1 -- 172.21.15.1:0/1472625801 learned_addr learned my addr 172.21.15.1:0/1472625801 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f859effd640 1 -- 172.21.15.1:0/960131947 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c031640 con 0x7f8598052480 2024-03-24T10:34:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 0x7fe4300b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fe435772640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 0x7fe43008ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f859effd640 1 -- 172.21.15.1:0/960131947 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858c0317a0 con 0x7f8598052480 2024-03-24T10:34:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7f859effd640 1 -- 172.21.15.1:0/960131947 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c031b30 con 0x7f8598052480 2024-03-24T10:34:02.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.720+0000 7fd6637fe640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd664143230 0x7fd6641505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe43016ba80 con 0x7fe41c005180 2024-03-24T10:34:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fe436774640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 0x7fe4300b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd664150110 con 0x7fd664143230 2024-03-24T10:34:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd66414f500 con 0x7fd664070b00 2024-03-24T10:34:02.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd66414f7e0 con 0x7fd66406ebf0 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f98f3fff640 1 -- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 msgr2=0x7f98f406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f98f3fff640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 0x7f98f406f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f98f3fff640 1 -- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 msgr2=0x7f98f40c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f98f3fff640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f98f3fff640 1 -- 172.21.15.1:0/1472625801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ec009b30 con 0x7f98f415df60 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe43016c450 con 0x7fe43016d6e0 2024-03-24T10:34:02.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe43016d1f0 con 0x7fe43016cd50 2024-03-24T10:34:02.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f43095c8640 1 -- 172.21.15.1:0/1144940669 shutdown_connections 2024-03-24T10:34:02.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a1531640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a1531640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24940e0010 0x7f2494007230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f43095c8640 1 -- 172.21.15.1:0/1144940669 wait complete. 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f249b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 -- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd664070b00 msgr2=0x7fd664135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd664070b00 0x7fd664135230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 -- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd664143230 msgr2=0x7fd6641505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd664143230 0x7fd6641505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6540094d0 con 0x7fd66406ebf0 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f249b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48240/0 (socket says 172.21.15.1:48240) 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a1531640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f249404ffd0 con 0x7f24940e0010 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a1531640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f249404f500 con 0x7f2494008f40 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a1531640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f249404d010 con 0x7f249403adf0 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f249b7fe640 1 -- 172.21.15.1:0/1604870651 learned_addr learned my addr 172.21.15.1:0/1604870651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f249bfff640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2494008f40 0x7f24940cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7f24a0d30640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24940e0010 0x7f2494007230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.721+0000 7fd662ffd640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd654004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 -- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598052480 msgr2=0x7f85980528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f98f3fff640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98e800c450 tx=0x7f98e800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598052480 0x7f85980528f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f858c009870 tx=0x7f858c02b8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 -- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2494008f40 msgr2=0x7f24940cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2494008f40 0x7f24940cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 -- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24940e0010 msgr2=0x7f2494007230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24940e0010 0x7f2494007230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 -- 172.21.15.1:0/1604870651 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24900094d0 con 0x7f249403adf0 2024-03-24T10:34:02.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654039d50 con 0x7fd66406ebf0 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd654043400 con 0x7fd66406ebf0 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 -- 172.21.15.1:0/960131947 shutdown_connections 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598052480 0x7f85980528f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 --2- 172.21.15.1:0/960131947 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598037680 0x7f8598057680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66414fac0 con 0x7fd66406ebf0 2024-03-24T10:34:02.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66414f140 con 0x7fd66406ebf0 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 -- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 msgr2=0x7fe4300b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 0x7fe4300b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 -- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 msgr2=0x7fe43016bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 -- 172.21.15.1:0/960131947 >> 172.21.15.1:0/960131947 conn(0x7f8598020030 msgr2=0x7f8598056d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe43016bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43016b430 con 0x7fe43016cd50 2024-03-24T10:34:02.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fef57fff640 1 Processor -- start 2024-03-24T10:34:02.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6540437b0 con 0x7fd66406ebf0 2024-03-24T10:34:02.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fef57fff640 1 -- start start 2024-03-24T10:34:02.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f249b7fe640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2488014170 tx=0x7f2488014630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe435772640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 0x7fe43008ab50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe42000d860 tx=0x7fe42000dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7fe436774640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 0x7fe4300b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:02.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e80052d0 con 0x7f98f415df60 2024-03-24T10:34:02.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24880114e0 con 0x7f249403adf0 2024-03-24T10:34:02.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2488016050 con 0x7f249403adf0 2024-03-24T10:34:02.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488014e60 con 0x7f249403adf0 2024-03-24T10:34:02.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249404f730 con 0x7f249403adf0 2024-03-24T10:34:02.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7f43095c8640 1 Processor -- start 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.722+0000 7f85a5286640 1 -- 172.21.15.1:0/960131947 shutdown_connections 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e8005430 con 0x7f98f415df60 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7f98f9d92640 1 -- 172.21.15.1:0/1472625801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f40c7c20 con 0x7f98f415df60 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7f98f9d92640 1 -- 172.21.15.1:0/1472625801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f415d740 con 0x7f98f415df60 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420008aa0 con 0x7fe43016cd50 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42000c040 con 0x7fe43016cd50 2024-03-24T10:34:02.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420008e00 con 0x7fe43016cd50 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43016b710 con 0x7fe43016cd50 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43016db90 con 0x7fe43016cd50 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.723+0000 7fef57fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef500d31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e8012830 con 0x7f98f415df60 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef57fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 0x7fef500da670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef57fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef500e1cc0 0x7fef500e2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef57fff640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fef500e2680 con 0x7fef50036ca0 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef57fff640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fef500d81d0 con 0x7fef500e1cc0 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef57fff640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fef500d84b0 con 0x7fef500df220 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f40c8140 con 0x7f98f415df60 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7f43095c8640 1 -- start start 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42fc006ee0 0x7f42fc036e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc04e250 0x7f42fc0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc0e1320 con 0x7f42fc006ee0 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc04cc70 con 0x7f42fc04ffc0 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43095c8640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc04cf50 con 0x7f42fc04e250 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7f85a5286640 1 -- 172.21.15.1:0/960131947 wait complete. 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f249404fcf0 con 0x7f249403adf0 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 0x7fef500da670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef567fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 0x7fef500da670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33740/0 (socket says 172.21.15.1:33740) 2024-03-24T10:34:02.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef567fc640 1 -- 172.21.15.1:0/3398419149 learned_addr learned my addr 172.21.15.1:0/3398419149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fef56ffd640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef500d31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.724+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd654043910 con 0x7fd66406ebf0 2024-03-24T10:34:02.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd644005290 con 0x7fd66406ebf0 2024-03-24T10:34:02.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe418005290 con 0x7fe43016cd50 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43037fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43037fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49888/0 (socket says 172.21.15.1:49888) 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f43037fe640 1 -- 172.21.15.1:0/3302590783 learned_addr learned my addr 172.21.15.1:0/3302590783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7f4303fff640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42fc006ee0 0x7f42fc036e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fd64bfff640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 0x7fd65006ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd65403e070 con 0x7fd66406ebf0 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe420012460 con 0x7fe43016cd50 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f43037fe640 1 -- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc04e250 msgr2=0x7f42fc0d0a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f43037fe640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc04e250 0x7f42fc0d0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f43037fe640 1 -- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42fc006ee0 msgr2=0x7f42fc036e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f43037fe640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42fc006ee0 0x7f42fc036e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f43037fe640 1 -- 172.21.15.1:0/3302590783 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f8009400 con 0x7f42fc04ffc0 2024-03-24T10:34:02.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f248800f850 con 0x7f249403adf0 2024-03-24T10:34:02.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7fd6627fc640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 0x7fd65006ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7fe4267fc640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 0x7fe41406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f43037fe640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42f000c3f0 tx=0x7f42f000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f85a5286640 1 Processor -- start 2024-03-24T10:34:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f2498ff9640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 0x7f248006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fef56ffd640 1 -- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef500e1cc0 msgr2=0x7fef500e2140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T10:34:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fef56ffd640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef500e1cc0 0x7fef500e2140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fef56ffd640 1 -- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 msgr2=0x7fef500da670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fef56ffd640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 0x7fef500da670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.725+0000 7fef56ffd640 1 -- 172.21.15.1:0/3398419149 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef500d8790 con 0x7fef500df220 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7fef56ffd640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef500d31d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef480118c0 tx=0x7fef48011cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.726+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f98e8012990 con 0x7f98f415df60 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f98f17fa640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 0x7f98e006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe42008a3d0 con 0x7fe43016cd50 2024-03-24T10:34:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7fe435f73640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 0x7fe41406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f85a5286640 1 -- start start 2024-03-24T10:34:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f249bfff640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 0x7f248006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f98e8060eb0 con 0x7f98f415df60 2024-03-24T10:34:02.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f98f8d90640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 0x7f98e006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2494005700 con 0x7f249403adf0 2024-03-24T10:34:02.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.727+0000 7f249bfff640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 0x7f248006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f249003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7fd6627fc640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 0x7fd65006ddd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd66415f770 tx=0x7fd65c007770 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8598037a60 0x7f85980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 0x7f85980cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f859ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f85980e3050 con 0x7f8598037a60 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f85980e36b0 con 0x7f8598037680 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a5286640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f85980e3990 con 0x7f8598052480 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f859ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49906/0 (socket says 172.21.15.1:49906) 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f859ffff640 1 -- 172.21.15.1:0/3926888028 learned_addr learned my addr 172.21.15.1:0/3926888028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7fe435f73640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 0x7fe41406ddd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006c80 tx=0x7fe42c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f98f8d90640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 0x7f98e006dd30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000f60 tx=0x7f98e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f859f7fe640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8598037a60 0x7f85980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.728+0000 7f85a4a85640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 0x7f85980cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 -- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 msgr2=0x7f85980cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 0x7f85980cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 -- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8598037a60 msgr2=0x7f85980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8598037a60 0x7f85980cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 -- 172.21.15.1:0/3926888028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c0094d0 con 0x7f8598037680 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f85a4a85640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 0x7f85980cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859ffff640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f85a0062cd0 tx=0x7f85a0068c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0090050 con 0x7f8598037680 2024-03-24T10:34:02.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.729+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f00074e0 con 0x7f42fc04ffc0 2024-03-24T10:34:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f0007640 con 0x7f42fc04ffc0 2024-03-24T10:34:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc04d230 con 0x7f42fc04ffc0 2024-03-24T10:34:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f85a5286640 1 -- 172.21.15.1:0/3926888028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85980e3330 con 0x7f8598037680 2024-03-24T10:34:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f85a5286640 1 -- 172.21.15.1:0/3926888028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85980cfee0 con 0x7f8598037680 2024-03-24T10:34:02.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0091040 con 0x7f8598037680 2024-03-24T10:34:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.730+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f98e8054a40 con 0x7f98f415df60 2024-03-24T10:34:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.731+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc04d7f0 con 0x7f42fc04ffc0 2024-03-24T10:34:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.731+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0004a00 con 0x7f42fc04ffc0 2024-03-24T10:34:02.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.731+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd654077b80 con 0x7fd66406ebf0 2024-03-24T10:34:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.731+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0071730 con 0x7f8598037680 2024-03-24T10:34:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.731+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8578005290 con 0x7f8598037680 2024-03-24T10:34:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.732+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f85a0071910 con 0x7f8598037680 2024-03-24T10:34:02.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.732+0000 7f859cff9640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 0x7f858406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.732+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42f0004c20 con 0x7f42fc04ffc0 2024-03-24T10:34:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.732+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f85a0108a10 con 0x7f8598037680 2024-03-24T10:34:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc04da00 con 0x7f42fc04ffc0 2024-03-24T10:34:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f859f7fe640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 0x7f858406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f4300ff9640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 0x7f42e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f4303fff640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 0x7f42e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f859f7fe640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 0x7f858406dcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f859803cd40 tx=0x7f85940090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.733+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f42f0089f30 con 0x7f42fc04ffc0 2024-03-24T10:34:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.734+0000 7f4303fff640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 0x7f42e806dcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42f8036000 tx=0x7f42f8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.735+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2488060d50 con 0x7f249403adf0 2024-03-24T10:34:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.735+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800f040 con 0x7fef500df220 2024-03-24T10:34:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.735+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef500d8a70 con 0x7fef500df220 2024-03-24T10:34:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.735+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef500dabb0 con 0x7fef500df220 2024-03-24T10:34:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.735+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe420054f10 con 0x7fe43016cd50 2024-03-24T10:34:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.736+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef500d8f90 con 0x7fef500df220 2024-03-24T10:34:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.737+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f85a00d35e0 con 0x7f8598037680 2024-03-24T10:34:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.740+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2488097060 con 0x7f249403adf0 2024-03-24T10:34:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.740+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef48004230 con 0x7fef500df220 2024-03-24T10:34:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.740+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800f780 con 0x7fef500df220 2024-03-24T10:34:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.741+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42f00548b0 con 0x7f42fc04ffc0 2024-03-24T10:34:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.742+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fef4800f8e0 con 0x7fef500df220 2024-03-24T10:34:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.742+0000 7fef3f7fe640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 0x7fef4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.742+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef4808a6b0 con 0x7fef500df220 2024-03-24T10:34:02.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.743+0000 7fef567fc640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 0x7fef4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.744+0000 7fef567fc640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 0x7fef4406dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fef50005d20 tx=0x7fef4c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:02.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:02.745+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fef48055140 con 0x7fef500df220 2024-03-24T10:34:03.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.094+0000 7fb9326b7640 1 -- 172.21.15.1:0/2130812003 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb910005540 con 0x7fb918008bd0 2024-03-24T10:34:03.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.094+0000 7fb921ffb640 1 -- 172.21.15.1:0/2130812003 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb91c054f30 con 0x7fb918008bd0 2024-03-24T10:34:03.096 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.097+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 msgr2=0x7fb9180807d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 0x7fb9180807d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0734a0 tx=0x7fb928030000 comp rx=0 tx=0).stop 2024-03-24T10:34:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 msgr2=0x7fb92c163fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00d9c0 tx=0x7fb91c00de80 comp rx=0 tx=0).stop 2024-03-24T10:34:03.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 shutdown_connections 2024-03-24T10:34:03.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb91807e310 0x7fb9180807d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c16d260 0x7fb92c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c0ab080 0x7fb92c16cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 --2- 172.21.15.1:0/2130812003 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb918008bd0 0x7fb92c163fc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.098+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 >> 172.21.15.1:0/2130812003 conn(0x7fb92c0b16a0 msgr2=0x7fb92c0afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.099+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 shutdown_connections 2024-03-24T10:34:03.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.099+0000 7fb9177fe640 1 -- 172.21.15.1:0/2130812003 wait complete. 2024-03-24T10:34:03.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-24T10:34:03.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.130+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd644005540 con 0x7fd66406ebf0 2024-03-24T10:34:03.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.131+0000 7fd64bfff640 1 -- 172.21.15.1:0/3707616343 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd654034020 con 0x7fd66406ebf0 2024-03-24T10:34:03.132 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:03.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.133+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 msgr2=0x7fd65006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.133+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 0x7fd65006ddd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd66415f770 tx=0x7fd65c007770 comp rx=0 tx=0).stop 2024-03-24T10:34:03.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 msgr2=0x7fd66414ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd654004730 comp rx=0 tx=0).stop 2024-03-24T10:34:03.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 shutdown_connections 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd65006b910 0x7fd65006ddd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd664143230 0x7fd6641505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd664070b00 0x7fd664135230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 --2- 172.21.15.1:0/3707616343 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd66406ebf0 0x7fd66414ec00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 >> 172.21.15.1:0/3707616343 conn(0x7fd66406b0a0 msgr2=0x7fd66414dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 shutdown_connections 2024-03-24T10:34:03.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.134+0000 7fd66996c640 1 -- 172.21.15.1:0/3707616343 wait complete. 2024-03-24T10:34:03.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.147+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f98f40c8140 con 0x7f98f415df60 2024-03-24T10:34:03.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.147+0000 7f98f17fa640 1 -- 172.21.15.1:0/1472625801 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f98f40c8140 con 0x7f98f415df60 2024-03-24T10:34:03.149 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:03.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.150+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 msgr2=0x7f98e006dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.150+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 0x7f98e006dd30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98e4000f60 tx=0x7f98e4009040 comp rx=0 tx=0).stop 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.150+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 msgr2=0x7f98f41f81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.150+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98e800c450 tx=0x7f98e800c910 comp rx=0 tx=0).stop 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 shutdown_connections 2024-03-24T10:34:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f98e006b870 0x7f98e006dd30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f98f41f8720 0x7f98f406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98f415df60 0x7f98f41f81e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 --2- 172.21.15.1:0/1472625801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98f40d2a40 0x7f98f40c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 >> 172.21.15.1:0/1472625801 conn(0x7f98f409d0c0 msgr2=0x7f98f40b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 shutdown_connections 2024-03-24T10:34:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.151+0000 7f98daffd640 1 -- 172.21.15.1:0/1472625801 wait complete. 2024-03-24T10:34:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.158+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f249403b920 con 0x7f249403adf0 2024-03-24T10:34:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.158+0000 7f2498ff9640 1 -- 172.21.15.1:0/1604870651 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f248805c8f0 con 0x7f249403adf0 2024-03-24T10:34:03.159 INFO:teuthology.orchestra.run.smithi001.stdout:55834574852 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 msgr2=0x7f248006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 0x7f248006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f249003c040 comp rx=0 tx=0).stop 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 msgr2=0x7f249404efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2488014170 tx=0x7f2488014630 comp rx=0 tx=0).stop 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 shutdown_connections 2024-03-24T10:34:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f248006b890 0x7f248006dd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24940e0010 0x7f2494007230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f249403adf0 0x7f249404efc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 --2- 172.21.15.1:0/1604870651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2494008f40 0x7f24940cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.160+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 >> 172.21.15.1:0/1604870651 conn(0x7f249401ef60 msgr2=0x7f2494006ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.161+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f42fc0052b0 con 0x7f42fc04ffc0 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.161+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 shutdown_connections 2024-03-24T10:34:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.161+0000 7f24a1531640 1 -- 172.21.15.1:0/1604870651 wait complete. 2024-03-24T10:34:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.161+0000 7f4300ff9640 1 -- 172.21.15.1:0/3302590783 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f42f0057f70 con 0x7f42fc04ffc0 2024-03-24T10:34:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.161+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe418005540 con 0x7fe43016cd50 2024-03-24T10:34:03.164 INFO:teuthology.orchestra.run.smithi001.stdout:55834574853 2024-03-24T10:34:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.163+0000 7fe4267fc640 1 -- 172.21.15.1:0/1209516742 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe4200585d0 con 0x7fe43016cd50 2024-03-24T10:34:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.163+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8578005810 con 0x7f8598037680 2024-03-24T10:34:03.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2024-03-24T10:34:03.166 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:03.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.163+0000 7f859cff9640 1 -- 172.21.15.1:0/3926888028 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85a00d6ca0 con 0x7f8598037680 2024-03-24T10:34:03.167 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.164+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 msgr2=0x7f42e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.164+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 0x7f42e806dcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42f8036000 tx=0x7f42f8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 msgr2=0x7f42fc04dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42f000c3f0 tx=0x7f42f000c8b0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 shutdown_connections 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42e806b810 0x7f42e806dcd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42fc04e250 0x7f42fc0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42fc04ffc0 0x7f42fc04dd10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 --2- 172.21.15.1:0/3302590783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42fc006ee0 0x7f42fc036e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 >> 172.21.15.1:0/3302590783 conn(0x7f42fc009fa0 msgr2=0x7f42fc049630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 shutdown_connections 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.165+0000 7f43095c8640 1 -- 172.21.15.1:0/3302590783 wait complete. 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.166+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 msgr2=0x7fe41406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.166+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 0x7fe41406ddd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006c80 tx=0x7fe42c008040 comp rx=0 tx=0).stop 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.166+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 msgr2=0x7fe43008ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.166+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 0x7fe43008ab50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe42000d860 tx=0x7fe42000dd20 comp rx=0 tx=0).stop 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 msgr2=0x7f858406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 0x7f858406dcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f859803cd40 tx=0x7f85940090e0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 msgr2=0x7f85980dd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f85a0062cd0 tx=0x7f85a0068c40 comp rx=0 tx=0).stop 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 shutdown_connections 2024-03-24T10:34:03.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe41406b910 0x7fe41406ddd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe43016d6e0 0x7fe4300b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe43016cd50 0x7fe43008ab50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 --2- 172.21.15.1:0/1209516742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe41c005180 0x7fe43016bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 >> 172.21.15.1:0/1209516742 conn(0x7fe4300b16a0 msgr2=0x7fe4300b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 shutdown_connections 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858406b810 0x7f858406dcd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8598052480 0x7f85980cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8598037a60 0x7f85980cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 --2- 172.21.15.1:0/3926888028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8598037680 0x7f85980dd590 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 >> 172.21.15.1:0/3926888028 conn(0x7f8598020030 msgr2=0x7f8598036660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 shutdown_connections 2024-03-24T10:34:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fe436f75640 1 -- 172.21.15.1:0/1209516742 wait complete. 2024-03-24T10:34:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 shutdown_connections 2024-03-24T10:34:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7f858a7fc640 1 -- 172.21.15.1:0/3926888028 wait complete. 2024-03-24T10:34:03.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fef500d8f90 con 0x7fef500df220 2024-03-24T10:34:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.167+0000 7fef3f7fe640 1 -- 172.21.15.1:0/3398419149 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef500d8f90 con 0x7fef500df220 2024-03-24T10:34:03.173 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 msgr2=0x7fef4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 0x7fef4406dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fef50005d20 tx=0x7fef4c009190 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 msgr2=0x7fef500d31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef500d31d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef480118c0 tx=0x7fef48011cc0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 shutdown_connections 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef4406b810 0x7fef4406dcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef500e1cc0 0x7fef500e2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef50036ca0 0x7fef500da670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 --2- 172.21.15.1:0/3398419149 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef500df220 0x7fef500d31d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 >> 172.21.15.1:0/3398419149 conn(0x7fef50009fb0 msgr2=0x7fef5004d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 shutdown_connections 2024-03-24T10:34:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:03.170+0000 7fef57fff640 1 -- 172.21.15.1:0/3398419149 wait complete. 2024-03-24T10:34:03.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-03-24T10:34:03.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-24T10:34:03.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.0 2024-03-24T10:34:03.188 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:03.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2024-03-24T10:34:03.189 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:03.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2024-03-24T10:34:03.190 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T10:34:04.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T10:34:04.167 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T10:34:04.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T10:34:04.183 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T10:34:04.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.238+0000 7fecc9684640 1 Processor -- start 2024-03-24T10:34:04.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.238+0000 7fecc9684640 1 -- start start 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc9684640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc9684640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecc40af950 0x7fecc40cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc9684640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecc40b3f70 con 0x7fecc406da60 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc9684640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fecc40cf850 con 0x7fecc40af950 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc9684640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fecc406b220 con 0x7fecc415a520 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48276/0 (socket says 172.21.15.1:48276) 2024-03-24T10:34:04.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc27fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecc40af950 0x7fecc40cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 -- 172.21.15.1:0/3070391488 learned_addr learned my addr 172.21.15.1:0/3070391488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 -- 172.21.15.1:0/3070391488 >> v2:172.21.15.1:3300/0 conn(0x7fecc40af950 msgr2=0x7fecc40cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 --2- 172.21.15.1:0/3070391488 >> v2:172.21.15.1:3300/0 conn(0x7fecc40af950 0x7fecc40cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 -- 172.21.15.1:0/3070391488 >> v1:172.21.15.3:6789/0 conn(0x7fecc406da60 legacy=0x7fecc406bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.239+0000 7fecc2ffd640 1 -- 172.21.15.1:0/3070391488 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc406b500 con 0x7fecc415a520 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.240+0000 7fecc2ffd640 1 --2- 172.21.15.1:0/3070391488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009870 tx=0x7fecb402db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52229940726da8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.240+0000 7fecc1ffb640 1 -- 172.21.15.1:0/3070391488 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb402e940 con 0x7fecc415a520 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.241+0000 7fecc1ffb640 1 -- 172.21.15.1:0/3070391488 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb402eaa0 con 0x7fecc415a520 2024-03-24T10:34:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.241+0000 7fecc1ffb640 1 -- 172.21.15.1:0/3070391488 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb402ee30 con 0x7fecc415a520 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.241+0000 7fecc9684640 1 -- 172.21.15.1:0/3070391488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 msgr2=0x7fecc40b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.241+0000 7fecc9684640 1 --2- 172.21.15.1:0/3070391488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009870 tx=0x7fecb402db90 comp rx=0 tx=0).stop 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 -- 172.21.15.1:0/3070391488 shutdown_connections 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 --2- 172.21.15.1:0/3070391488 >> v2:172.21.15.1:3300/0 conn(0x7fecc40af950 0x7fecc40cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 --2- 172.21.15.1:0/3070391488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc415a520 0x7fecc40b3a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 -- 172.21.15.1:0/3070391488 >> 172.21.15.1:0/3070391488 conn(0x7fecc40ccec0 msgr2=0x7fecc40cd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 -- 172.21.15.1:0/3070391488 shutdown_connections 2024-03-24T10:34:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.242+0000 7fecc9684640 1 -- 172.21.15.1:0/3070391488 wait complete. 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.243+0000 7fecc9684640 1 Processor -- start 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.243+0000 7fecc9684640 1 -- start start 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc40af950 0x7fecc4146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecc415a520 0x7fecc4146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33770/0 (socket says 172.21.15.1:33770) 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4157710 con 0x7fecc406da60 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fecc41f3a60 con 0x7fecc415a520 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc9684640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fecc4156e70 con 0x7fecc40af950 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc2ffd640 1 -- 172.21.15.1:0/2766627949 learned_addr learned my addr 172.21.15.1:0/2766627949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.244+0000 7fecc27fc640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc40af950 0x7fecc4146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 -- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc40af950 msgr2=0x7fecc4146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc40af950 0x7fecc4146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 -- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecc415a520 msgr2=0x7fecc4146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecc415a520 0x7fecc4146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 -- 172.21.15.1:0/2766627949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb40094d0 con 0x7fecc406da60 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc2ffd640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009870 tx=0x7fecb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4041680 con 0x7fecc406da60 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb40417e0 con 0x7fecc406da60 2024-03-24T10:34:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4041b50 con 0x7fecc406da60 2024-03-24T10:34:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.245+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc4157150 con 0x7fecc406da60 2024-03-24T10:34:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.246+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc415ada0 con 0x7fecc406da60 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.247+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc406e8b0 con 0x7fecc406da60 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.248+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fecb4041cb0 con 0x7fecc406da60 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.248+0000 7fecab7fe640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 0x7fecb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.248+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fecb40acf10 con 0x7fecc406da60 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.248+0000 7fecc27fc640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 0x7fecb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.249+0000 7fecc27fc640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 0x7fecb006dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fecc4156180 tx=0x7fecb8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.253+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fecb4077930 con 0x7fecc406da60 2024-03-24T10:34:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.310+0000 7f82022c0640 1 Processor -- start 2024-03-24T10:34:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.310+0000 7f82022c0640 1 -- start start 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.311+0000 7f82022c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 0x7f81fc06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.311+0000 7f82022c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc070710 con 0x7f81fc15f260 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.311+0000 7f82022c0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc06e1e0 con 0x7f81fc15de70 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.311+0000 7f82022c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc06ec30 con 0x7f81fc06b7b0 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f81fb7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f81fc15f260 0x7f81fc06b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41650/0 (socket says 172.21.15.1:41650) 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f81fb7fe640 1 -- 172.21.15.1:0/3633709817 learned_addr learned my addr 172.21.15.1:0/3633709817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f820085d640 1 --2- 172.21.15.1:0/3633709817 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 0x7f81fc06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f81fbfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81fc15de70 0x7f81fc1625d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55418/0 (socket says 172.21.15.1:55418) 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f820085d640 1 -- 172.21.15.1:0/3633709817 >> v1:172.21.15.3:6789/0 conn(0x7f81fc15de70 legacy=0x7f81fc1625d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f820085d640 1 -- 172.21.15.1:0/3633709817 >> v1:172.21.15.1:6789/0 conn(0x7f81fc15f260 legacy=0x7f81fc06b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.312+0000 7f820085d640 1 -- 172.21.15.1:0/3633709817 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc0701c0 con 0x7f81fc06b7b0 2024-03-24T10:34:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.313+0000 7f820085d640 1 --2- 172.21.15.1:0/3633709817 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 0x7f81fc06e6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009dc0 tx=0x7f81f402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ded3ba76c574d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.313+0000 7f81faffd640 1 -- 172.21.15.1:0/3633709817 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f402cc40 con 0x7f81fc06b7b0 2024-03-24T10:34:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.313+0000 7f81faffd640 1 -- 172.21.15.1:0/3633709817 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f402cda0 con 0x7f81fc06b7b0 2024-03-24T10:34:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.313+0000 7f81faffd640 1 -- 172.21.15.1:0/3633709817 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4034660 con 0x7f81fc06b7b0 2024-03-24T10:34:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 -- 172.21.15.1:0/3633709817 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 msgr2=0x7f81fc06e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 --2- 172.21.15.1:0/3633709817 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 0x7f81fc06e6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009dc0 tx=0x7f81f402be90 comp rx=0 tx=0).stop 2024-03-24T10:34:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 -- 172.21.15.1:0/3633709817 shutdown_connections 2024-03-24T10:34:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 --2- 172.21.15.1:0/3633709817 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc06b7b0 0x7f81fc06e6f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 -- 172.21.15.1:0/3633709817 >> 172.21.15.1:0/3633709817 conn(0x7f81fc0d0c20 msgr2=0x7f81fc0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 -- 172.21.15.1:0/3633709817 shutdown_connections 2024-03-24T10:34:04.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.314+0000 7f82022c0640 1 -- 172.21.15.1:0/3633709817 wait complete. 2024-03-24T10:34:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.316+0000 7f82022c0640 1 Processor -- start 2024-03-24T10:34:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.317+0000 7f82022c0640 1 -- start start 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc06b7b0 0x7f81fc150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81fc15f260 0x7f81fc152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc0afad0 con 0x7f81fc06b7b0 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc06b7b0 0x7f81fc150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc156b70 con 0x7f81fc15f260 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f82022c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc156e50 con 0x7f81fc15de70 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48298/0 (socket says 172.21.15.1:48298) 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 -- 172.21.15.1:0/1606968400 learned_addr learned my addr 172.21.15.1:0/1606968400 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f820085d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81fc15f260 0x7f81fc152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 -- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81fc15f260 msgr2=0x7f81fc152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81fc15f260 0x7f81fc152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 -- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc06b7b0 msgr2=0x7f81fc150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc06b7b0 0x7f81fc150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.318+0000 7f81fb7fe640 1 -- 172.21.15.1:0/1606968400 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f4009a70 con 0x7f81fc15de70 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f81fb7fe640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81f000c3c0 tx=0x7f81f000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0005ce0 con 0x7f81fc15de70 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f0005e40 con 0x7f81fc15de70 2024-03-24T10:34:04.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc157130 con 0x7f81fc15de70 2024-03-24T10:34:04.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f00072d0 con 0x7f81fc15de70 2024-03-24T10:34:04.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.319+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc14a5b0 con 0x7f81fc15de70 2024-03-24T10:34:04.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.321+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc073590 con 0x7f81fc15de70 2024-03-24T10:34:04.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.321+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f81f00074f0 con 0x7f81fc15de70 2024-03-24T10:34:04.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.322+0000 7f81f8ff9640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 0x7f81e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.322+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f81f0089610 con 0x7f81fc15de70 2024-03-24T10:34:04.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.322+0000 7f81fbfff640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 0x7f81e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.322+0000 7f81fbfff640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 0x7f81e806dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000f60 tx=0x7f81ec000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.324+0000 7f430008f640 1 Processor -- start 2024-03-24T10:34:04.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.324+0000 7f430008f640 1 -- start start 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f42f806bab0 0x7f42f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 0x7f42f806be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806b590 0x7f42f80b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f42f806f630 con 0x7f42f80cdf30 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f42f8073670 con 0x7f42f806bab0 2024-03-24T10:34:04.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f430008f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42f806e830 con 0x7f42f806b590 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f42fd62a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 0x7f42f806be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f42fde2b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f42f806bab0 0x7f42f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f42fd62a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 0x7f42f806be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33788/0 (socket says 172.21.15.1:33788) 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f42fd62a640 1 -- 172.21.15.1:0/1841616882 learned_addr learned my addr 172.21.15.1:0/1841616882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.325+0000 7f42fe62c640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806b590 0x7f42f80b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.326+0000 7f42fde2b640 1 -- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806b590 msgr2=0x7f42f80b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.326+0000 7f42fde2b640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806b590 0x7f42f80b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.326+0000 7f42fde2b640 1 -- 172.21.15.1:0/1841616882 >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 msgr2=0x7f42f806be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.326+0000 7f42fde2b640 1 --2- 172.21.15.1:0/1841616882 >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 0x7f42f806be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.326+0000 7f42fde2b640 1 -- 172.21.15.1:0/1841616882 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f806c3d0 con 0x7f42f806bab0 2024-03-24T10:34:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.327+0000 7f42fde2b640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806bab0 0x7f42f806e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42f8072760 tx=0x7f42f402cd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ee1403270f729e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.327+0000 7f42fce29640 1 -- 172.21.15.1:0/1841616882 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f402dab0 con 0x7f42f806bab0 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.327+0000 7f42fce29640 1 -- 172.21.15.1:0/1841616882 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f402dc10 con 0x7f42f806bab0 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.327+0000 7f42fce29640 1 -- 172.21.15.1:0/1841616882 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4035440 con 0x7f42f806bab0 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 -- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806bab0 msgr2=0x7f42f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806bab0 0x7f42f806e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42f8072760 tx=0x7f42f402cd00 comp rx=0 tx=0).stop 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 -- 172.21.15.1:0/1841616882 shutdown_connections 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806b590 0x7f42f80b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 --2- 172.21.15.1:0/1841616882 >> v2:172.21.15.1:3300/0 conn(0x7f42f80cdf30 0x7f42f806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 --2- 172.21.15.1:0/1841616882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806bab0 0x7f42f806e2f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 -- 172.21.15.1:0/1841616882 >> 172.21.15.1:0/1841616882 conn(0x7f42f8072850 msgr2=0x7f42f8072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 -- 172.21.15.1:0/1841616882 shutdown_connections 2024-03-24T10:34:04.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f81f00540d0 con 0x7f81fc15de70 2024-03-24T10:34:04.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.328+0000 7f430008f640 1 -- 172.21.15.1:0/1841616882 wait complete. 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.329+0000 7f430008f640 1 Processor -- start 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.329+0000 7f430008f640 1 -- start start 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 0x7f42f8159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 0x7f42f815d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f80cdf30 0x7f42f80c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42f80c4ee0 con 0x7f42f806bab0 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f42f815c1e0 con 0x7f42f806b590 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f430008f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42f815c4c0 con 0x7f42f80cdf30 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.330+0000 7f42fde2b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 0x7f42f8159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fe62c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f80cdf30 0x7f42f80c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 0x7f42f815d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fde2b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 0x7f42f8159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49940/0 (socket says 172.21.15.1:49940) 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fde2b640 1 -- 172.21.15.1:0/3442940850 learned_addr learned my addr 172.21.15.1:0/3442940850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 -- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f80cdf30 msgr2=0x7f42f80c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f80cdf30 0x7f42f80c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 -- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 msgr2=0x7f42f8159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 0x7f42f8159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 -- 172.21.15.1:0/3442940850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f400a5a0 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42fd62a640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 0x7f42f815d3e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e930 tx=0x7f42e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.331+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e800c7f0 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.332+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e800c950 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.332+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e800cd00 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.332+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f815c7a0 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.332+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f815be20 con 0x7f42f806bab0 2024-03-24T10:34:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.333+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f80c5090 con 0x7f42f806bab0 2024-03-24T10:34:04.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.335+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42e8013470 con 0x7f42f806bab0 2024-03-24T10:34:04.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.335+0000 7f42ee7fc640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 0x7f42dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.335+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f42e808a740 con 0x7f42f806bab0 2024-03-24T10:34:04.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.336+0000 7f42fde2b640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 0x7f42dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.337+0000 7f42fde2b640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 0x7f42dc06dd00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42f402cd40 tx=0x7f42f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.338+0000 7f9ebad3a640 1 Processor -- start 2024-03-24T10:34:04.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9ebad3a640 1 -- start start 2024-03-24T10:34:04.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9ebad3a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 0x7f9eb40ce950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9ebad3a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9eb406b7b0 0x7f9eb40cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9ebad3a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eb415f490 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9ebad3a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9eb40ca810 con 0x7f9eb406b7b0 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9ebad3a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb40caaf0 con 0x7f9eb415f0b0 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9eb8ad6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 0x7f9eb40ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.339+0000 7f9eb3fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9eb40c9ea0 0x7f9eb406b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55430/0 (socket says 172.21.15.1:55430) 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb3fff640 1 -- 172.21.15.1:0/1962010654 learned_addr learned my addr 172.21.15.1:0/1962010654 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb8ad6640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 0x7f9eb40ce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48330/0 (socket says 172.21.15.1:48330) 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb92d7640 1 --2- 172.21.15.1:0/1962010654 >> v2:172.21.15.3:3300/0 conn(0x7f9eb406b7b0 0x7f9eb40cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb37fe640 1 -- 172.21.15.1:0/1962010654 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2576749233 0 0) 0x7f9eb415f490 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb37fe640 1 -- 172.21.15.1:0/1962010654 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ea4003520 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb92d7640 1 -- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 msgr2=0x7f9eb40ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb92d7640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 0x7f9eb40ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb92d7640 1 -- 172.21.15.1:0/1962010654 >> v1:172.21.15.3:6789/0 conn(0x7f9eb40c9ea0 legacy=0x7f9eb406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.340+0000 7f9eb92d7640 1 -- 172.21.15.1:0/1962010654 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb40cadd0 con 0x7f9eb406b7b0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.341+0000 7f9eb92d7640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb40cc790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009d90 tx=0x7f9eac02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b039aa3172c548c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.341+0000 7f9eb37fe640 1 -- 172.21.15.1:0/1962010654 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac02cc80 con 0x7f9eb406b7b0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.341+0000 7f9eb37fe640 1 -- 172.21.15.1:0/1962010654 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eac02cde0 con 0x7f9eb406b7b0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.341+0000 7f9eb37fe640 1 -- 172.21.15.1:0/1962010654 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac034660 con 0x7f9eb406b7b0 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 -- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 msgr2=0x7f9eb40cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb40cc790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009d90 tx=0x7f9eac02bed0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 -- 172.21.15.1:0/1962010654 shutdown_connections 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb40cc790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/1962010654 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb415f0b0 0x7f9eb40ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 -- 172.21.15.1:0/1962010654 >> 172.21.15.1:0/1962010654 conn(0x7f9eb40d0b70 msgr2=0x7f9eb40d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 -- 172.21.15.1:0/1962010654 shutdown_connections 2024-03-24T10:34:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.342+0000 7f9ebad3a640 1 -- 172.21.15.1:0/1962010654 wait complete. 2024-03-24T10:34:04.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.343+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42e8055340 con 0x7f42f806bab0 2024-03-24T10:34:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.344+0000 7f9ebad3a640 1 Processor -- start 2024-03-24T10:34:04.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.345+0000 7f9ebad3a640 1 -- start start 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb4157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 0x7f9eb40b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 0x7f9eb40b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4155c80 con 0x7f9eb415f0b0 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb40b4fc0 con 0x7f9eb406b7b0 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9ebad3a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb40b52a0 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb8ad6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb4157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb8ad6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb4157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49946/0 (socket says 172.21.15.1:49946) 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb8ad6640 1 -- 172.21.15.1:0/68028045 learned_addr learned my addr 172.21.15.1:0/68028045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 0x7f9eb40b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb92d7640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 0x7f9eb40b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 -- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 msgr2=0x7f9eb4157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb4157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 -- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 msgr2=0x7f9eb40b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 0x7f9eb40b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.346+0000 7f9eb3fff640 1 -- 172.21.15.1:0/68028045 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac009a70 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9eb92d7640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 0x7f9eb40b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9eb3fff640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 0x7f9eb40b4490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c430 tx=0x7f9ea000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f299e702640 1 Processor -- start 2024-03-24T10:34:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0005ce0 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f299e702640 1 -- start start 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea0005e40 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00072d0 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb40b5580 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.347+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb40b6480 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb40b6610 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f29977fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f29980af860 0x7f299806b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41666/0 (socket says 172.21.15.1:41666) 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2997fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f299815f610 0x7f29980d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55438/0 (socket says 172.21.15.1:55438) 2024-03-24T10:34:04.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f299e702640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f299806b9c0 0x7f2998167000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f29977fe640 1 -- 172.21.15.1:0/4215032844 learned_addr learned my addr 172.21.15.1:0/4215032844 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:04.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ea0007430 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f299cc9f640 1 --2- 172.21.15.1:0/4215032844 >> v2:172.21.15.1:3300/0 conn(0x7f299806b9c0 0x7f2998167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29980ccd00 con 0x7f29980af860 2024-03-24T10:34:04.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29980ccfe0 con 0x7f299815f610 2024-03-24T10:34:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f29980cd2c0 con 0x7f299806b9c0 2024-03-24T10:34:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988512844 0 0) 0x7f29980ccd00 con 0x7f29980af860 2024-03-24T10:34:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2980003520 con 0x7f29980af860 2024-03-24T10:34:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f9eb17fa640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 0x7f9e9806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3976345745 0 0) 0x7f2980003520 con 0x7f29980af860 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 >> v2:172.21.15.1:3300/0 conn(0x7f299806b9c0 msgr2=0x7f2998167000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 --2- 172.21.15.1:0/4215032844 >> v2:172.21.15.1:3300/0 conn(0x7f299806b9c0 0x7f2998167000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 >> v1:172.21.15.3:6789/0 conn(0x7f299815f610 legacy=0x7f29980d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29980cd5a0 con 0x7f29980af860 2024-03-24T10:34:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f298c001670 con 0x7f29980af860 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 >> v1:172.21.15.1:6789/0 conn(0x7f29980af860 legacy=0x7f299806b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980003970 0x7f2980003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29800047c0 0x7f2980004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 0x7f298000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29980cd2c0 con 0x7f29800047c0 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f298000c080 con 0x7f2980005180 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f298000c240 con 0x7f2980003970 2024-03-24T10:34:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.349+0000 7f9eb8ad6640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 0x7f9e9806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 0x7f298000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f29977fe640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29800047c0 0x7f2980004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 -- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980003970 msgr2=0x7f2980003d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980003970 0x7f2980003d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 -- 172.21.15.1:0/4215032844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29800047c0 msgr2=0x7f2980004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29800047c0 0x7f2980004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f299cc9f640 1 -- 172.21.15.1:0/4215032844 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2980003520 con 0x7f2980005180 2024-03-24T10:34:04.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f29977fe640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29800047c0 0x7f2980004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:04.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9ea00897b0 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.350+0000 7f9eb8ad6640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 0x7f9e9806dda0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009b60 tx=0x7f9ea8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.351+0000 7f299cc9f640 1 -- 172.21.15.1:0/4215032844 reap_dead start 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.351+0000 7f299cc9f640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 0x7f298000b980 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f29900124e0 tx=0x7f29800057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8b37487d8979fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.351+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990003b90 con 0x7f2980005180 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.351+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990033050 con 0x7f2980005180 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.351+0000 7f2996ffd640 1 -- 172.21.15.1:0/4215032844 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990003b90 con 0x7f2980005180 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 msgr2=0x7f298000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 0x7f298000b980 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f29900124e0 tx=0x7f29800057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 shutdown_connections 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 --2- 172.21.15.1:0/4215032844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2980005180 0x7f298000b980 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 >> 172.21.15.1:0/4215032844 conn(0x7f2998072850 msgr2=0x7f2998072c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 shutdown_connections 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.352+0000 7f299e702640 1 -- 172.21.15.1:0/4215032844 wait complete. 2024-03-24T10:34:04.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.354+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ea0054260 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.354+0000 7f299e702640 1 Processor -- start 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.354+0000 7f299e702640 1 -- start start 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.355+0000 7f299e702640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299e702640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:48348/0 (socket says 172.21.15.1:48348) 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f29977fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f29977fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33816/0 (socket says 172.21.15.1:33816) 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 -- 172.21.15.1:0/1807340896 learned_addr learned my addr 172.21.15.1:0/1807340896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f299815d080 0x7f299815d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29980cbd20 con 0x7f29980cb360 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f299815e670 con 0x7f299815d080 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2998156b50 con 0x7f2980005180 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f299cc9f640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f299815d080 0x7f299815d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 -- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f299815d080 msgr2=0x7f299815d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f299815d080 0x7f299815d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 -- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 msgr2=0x7f29980cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.356+0000 7f2997fff640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299815ddd0 con 0x7f2980005180 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f29977fe640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f2997fff640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f298800c780 tx=0x7f298800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880072d0 con 0x7f2980005180 2024-03-24T10:34:04.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988007450 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880047e0 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299815e0b0 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.357+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29980b15b0 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.359+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299804f4e0 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.359+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2988004940 con 0x7f2980005180 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.359+0000 7f2994ff9640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 0x7f298406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.360+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2988089750 con 0x7f2980005180 2024-03-24T10:34:04.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.360+0000 7f29977fe640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 0x7f298406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.360+0000 7f29977fe640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 0x7f298406dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29981597b0 tx=0x7f298c007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.363+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2988054370 con 0x7f2980005180 2024-03-24T10:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.474+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fecc404f4e0 con 0x7fecc406da60 2024-03-24T10:34:04.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.474+0000 7fecab7fe640 1 -- 172.21.15.1:0/2766627949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fecb403b090 con 0x7fecc406da60 2024-03-24T10:34:04.474 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 msgr2=0x7fecb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 0x7fecb006dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fecc4156180 tx=0x7fecb8009190 comp rx=0 tx=0).stop 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 msgr2=0x7fecc41f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009870 tx=0x7fecb40046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 shutdown_connections 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fecb006b810 0x7fecb006dcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecc415a520 0x7fecc4146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fecc40af950 0x7fecc4146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 --2- 172.21.15.1:0/2766627949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecc406da60 0x7fecc41f3520 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.476+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 >> 172.21.15.1:0/2766627949 conn(0x7fecc40ccec0 msgr2=0x7fecc40714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.477+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 shutdown_connections 2024-03-24T10:34:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.477+0000 7fecc9684640 1 -- 172.21.15.1:0/2766627949 wait complete. 2024-03-24T10:34:04.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2024-03-24T10:34:04.491 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.522+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f81fc156f90 con 0x7f81fc15de70 2024-03-24T10:34:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.523+0000 7f81f8ff9640 1 -- 172.21.15.1:0/1606968400 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f81fc156f90 con 0x7f81fc15de70 2024-03-24T10:34:04.523 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:04.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 msgr2=0x7f81e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 0x7f81e806dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000f60 tx=0x7f81ec000ed0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 msgr2=0x7f81fc14a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81f000c3c0 tx=0x7f81f000c880 comp rx=0 tx=0).stop 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 shutdown_connections 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f81e806b810 0x7f81e806dcd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81fc15f260 0x7f81fc152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81fc15de70 0x7f81fc14a070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 --2- 172.21.15.1:0/1606968400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc06b7b0 0x7f81fc150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 >> 172.21.15.1:0/1606968400 conn(0x7f81fc0d0c20 msgr2=0x7f81fc0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 shutdown_connections 2024-03-24T10:34:04.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.525+0000 7f82022c0640 1 -- 172.21.15.1:0/1606968400 wait complete. 2024-03-24T10:34:04.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.1 2024-03-24T10:34:04.536 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.540+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9eb404f4e0 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.540+0000 7f9eb17fa640 1 -- 172.21.15.1:0/68028045 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9ea0057920 con 0x7f9eb40c9ea0 2024-03-24T10:34:04.541 INFO:teuthology.orchestra.run.smithi001.stdout:55834574854 2024-03-24T10:34:04.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 msgr2=0x7f9e9806dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 0x7f9e9806dda0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009b60 tx=0x7f9ea8009190 comp rx=0 tx=0).stop 2024-03-24T10:34:04.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 msgr2=0x7f9eb40b4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 0x7f9eb40b4490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c430 tx=0x7f9ea000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 shutdown_connections 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e9806b8e0 0x7f9e9806dda0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9eb415f0b0 0x7f9eb40b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9eb40c9ea0 0x7f9eb40b4490 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 --2- 172.21.15.1:0/68028045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9eb406b7b0 0x7f9eb4157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 >> 172.21.15.1:0/68028045 conn(0x7f9eb40d0b70 msgr2=0x7f9eb40ce670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.542+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 shutdown_connections 2024-03-24T10:34:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.543+0000 7f9ebad3a640 1 -- 172.21.15.1:0/68028045 wait complete. 2024-03-24T10:34:04.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2024-03-24T10:34:04.554 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.638+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f42f804f4e0 con 0x7f42f806bab0 2024-03-24T10:34:04.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.638+0000 7f42ee7fc640 1 -- 172.21.15.1:0/3442940850 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f42e8017020 con 0x7f42f806bab0 2024-03-24T10:34:04.638 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 msgr2=0x7f42dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 0x7f42dc06dd00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42f402cd40 tx=0x7f42f4008040 comp rx=0 tx=0).stop 2024-03-24T10:34:04.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 msgr2=0x7f42f815d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 0x7f42f815d3e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e930 tx=0x7f42e800edf0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 shutdown_connections 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42dc06b840 0x7f42dc06dd00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f80cdf30 0x7f42f80c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f806bab0 0x7f42f815d3e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 --2- 172.21.15.1:0/3442940850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f806b590 0x7f42f8159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 >> 172.21.15.1:0/3442940850 conn(0x7f42f8072850 msgr2=0x7f42f8072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 shutdown_connections 2024-03-24T10:34:04.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.640+0000 7f430008f640 1 -- 172.21.15.1:0/3442940850 wait complete. 2024-03-24T10:34:04.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2024-03-24T10:34:04.651 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.655+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2998073680 con 0x7f2980005180 2024-03-24T10:34:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.655+0000 7f2994ff9640 1 -- 172.21.15.1:0/1807340896 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2988018020 con 0x7f2980005180 2024-03-24T10:34:04.656 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 msgr2=0x7f298406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 0x7f298406dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29981597b0 tx=0x7f298c007040 comp rx=0 tx=0).stop 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 msgr2=0x7f2998156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f298800c780 tx=0x7f298800cc40 comp rx=0 tx=0).stop 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 shutdown_connections 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f298406b810 0x7f298406dcd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f299815d080 0x7f299815d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29980cb360 0x7f29980cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 --2- 172.21.15.1:0/1807340896 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2980005180 0x7f2998156610 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 >> 172.21.15.1:0/1807340896 conn(0x7f2998072850 msgr2=0x7f29980b5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 shutdown_connections 2024-03-24T10:34:04.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.657+0000 7f299e702640 1 -- 172.21.15.1:0/1807340896 wait complete. 2024-03-24T10:34:04.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2024-03-24T10:34:04.668 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:04.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T10:34:04.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:34:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.810+0000 7f5e5adb5640 1 Processor -- start 2024-03-24T10:34:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.810+0000 7f5e5adb5640 1 -- start start 2024-03-24T10:34:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e5adb5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e5406f0d0 0x7f5e5406f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e5adb5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5e5406e390 0x7f5e5406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e5adb5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e540b2f30 con 0x7f5e54146ee0 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e5adb5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f5e5406fa50 con 0x7f5e5406f0d0 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e5adb5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f5e5406ece0 con 0x7f5e5406e390 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e5406f0d0 0x7f5e5406f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5e5406f0d0 0x7f5e5406f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33830/0 (socket says 172.21.15.1:33830) 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 -- 172.21.15.1:0/2122247368 learned_addr learned my addr 172.21.15.1:0/2122247368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e59352640 1 --2- 172.21.15.1:0/2122247368 >> v2:172.21.15.3:3300/0 conn(0x7f5e5406e390 0x7f5e5406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 -- 172.21.15.1:0/2122247368 >> v2:172.21.15.3:3300/0 conn(0x7f5e5406e390 msgr2=0x7f5e5406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:04.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 --2- 172.21.15.1:0/2122247368 >> v2:172.21.15.3:3300/0 conn(0x7f5e5406e390 0x7f5e5406e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 -- 172.21.15.1:0/2122247368 >> v1:172.21.15.3:6789/0 conn(0x7f5e54146ee0 legacy=0x7f5e54071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e537fe640 1 -- 172.21.15.1:0/2122247368 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468564936 0 0) 0x7f5e540b2f30 con 0x7f5e54146ee0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.811+0000 7f5e53fff640 1 -- 172.21.15.1:0/2122247368 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e540b3180 con 0x7f5e5406f0d0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e53fff640 1 --2- 172.21.15.1:0/2122247368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e5406f510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e540afb30 tx=0x7f5e4002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484fff8956e4495f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e537fe640 1 -- 172.21.15.1:0/2122247368 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e400317e0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e537fe640 1 -- 172.21.15.1:0/2122247368 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e40031940 con 0x7f5e5406f0d0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e537fe640 1 -- 172.21.15.1:0/2122247368 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40031cd0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 -- 172.21.15.1:0/2122247368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 msgr2=0x7f5e5406f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/2122247368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e5406f510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e540afb30 tx=0x7f5e4002b880 comp rx=0 tx=0).stop 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 -- 172.21.15.1:0/2122247368 shutdown_connections 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/2122247368 >> v2:172.21.15.3:3300/0 conn(0x7f5e5406e390 0x7f5e5406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/2122247368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e5406f510 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.812+0000 7f5e5adb5640 1 -- 172.21.15.1:0/2122247368 >> 172.21.15.1:0/2122247368 conn(0x7f5e540cff00 msgr2=0x7f5e540d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.813+0000 7f5e5adb5640 1 -- 172.21.15.1:0/2122247368 shutdown_connections 2024-03-24T10:34:04.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.813+0000 7f5e5adb5640 1 -- 172.21.15.1:0/2122247368 wait complete. 2024-03-24T10:34:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.814+0000 7f5e5adb5640 1 Processor -- start 2024-03-24T10:34:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.814+0000 7f5e5adb5640 1 -- start start 2024-03-24T10:34:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e5406e390 0x7f5e540c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e54146ee0 0x7f5e540ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e540c5f40 con 0x7f5e5406f0d0 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5e540c50b0 con 0x7f5e54146ee0 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5e540c68a0 con 0x7f5e5406e390 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33840/0 (socket says 172.21.15.1:33840) 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 -- 172.21.15.1:0/3691842278 learned_addr learned my addr 172.21.15.1:0/3691842278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e58b51640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e5406e390 0x7f5e540c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e59352640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e54146ee0 0x7f5e540ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 -- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e5406e390 msgr2=0x7f5e540c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e5406e390 0x7f5e540c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 -- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e54146ee0 msgr2=0x7f5e540ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e54146ee0 0x7f5e540ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 -- 172.21.15.1:0/3691842278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e400094d0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e53fff640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002b8c0 tx=0x7f5e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40032880 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e40032a00 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e540c52c0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40032db0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.815+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e540c5650 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.816+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e540c6080 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.817+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e4004e450 con 0x7f5e5406f0d0 2024-03-24T10:34:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.817+0000 7f5e517fa640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 0x7f5e3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.817+0000 7f5e58b51640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 0x7f5e3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.817+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e400ac0d0 con 0x7f5e5406f0d0 2024-03-24T10:34:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.818+0000 7f5e58b51640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 0x7f5e3806dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e48000f60 tx=0x7f5e48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.822+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e540c6080 con 0x7f5e5406f0d0 2024-03-24T10:34:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:04.998+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e540b3a20 con 0x7f5e3806b810 2024-03-24T10:34:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.004+0000 7f5e517fa640 1 -- 172.21.15.1:0/3691842278 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f5e48001420 con 0x7f5e3806b810 2024-03-24T10:34:05.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:34:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 msgr2=0x7f5e3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 0x7f5e3806dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e48000f60 tx=0x7f5e48009040 comp rx=0 tx=0).stop 2024-03-24T10:34:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 msgr2=0x7f5e541e83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002b8c0 tx=0x7f5e400046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 shutdown_connections 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e3806b810 0x7f5e3806dcd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e54146ee0 0x7f5e540ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e5406f0d0 0x7f5e541e83d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 --2- 172.21.15.1:0/3691842278 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e5406e390 0x7f5e540c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 >> 172.21.15.1:0/3691842278 conn(0x7f5e540cff00 msgr2=0x7f5e540afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 shutdown_connections 2024-03-24T10:34:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.007+0000 7f5e5adb5640 1 -- 172.21.15.1:0/3691842278 wait complete. 2024-03-24T10:34:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-24T10:34:04.519712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235752,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738968,"statfs":{"total":773094113280,"available":772852703232,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.399262"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306747+0000","last_change":"2024-03-24T10:33:55.160859+0000","last_active":"2024-03-24T10:33:56.306747+0000","last_peered":"2024-03-24T10:33:56.306747+0000","last_clean":"2024-03-24T10:33:56.306747+0000","last_became_active":"2024-03-24T10:33:55.160069+0000","last_became_peered":"2024-03-24T10:33:55.160069+0000","last_unstale":"2024-03-24T10:33:56.306747+0000","last_undegraded":"2024-03-24T10:33:56.306747+0000","last_fullsized":"2024-03-24T10:33:56.306747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123102+0000","last_change":"2024-03-24T10:33:55.140972+0000","last_active":"2024-03-24T10:33:56.123102+0000","last_peered":"2024-03-24T10:33:56.123102+0000","last_clean":"2024-03-24T10:33:56.123102+0000","last_became_active":"2024-03-24T10:33:55.140573+0000","last_became_peered":"2024-03-24T10:33:55.140573+0000","last_unstale":"2024-03-24T10:33:56.123102+0000","last_undegraded":"2024-03-24T10:33:56.123102+0000","last_fullsized":"2024-03-24T10:33:56.123102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305754+0000","last_change":"2024-03-24T10:33:55.141078+0000","last_active":"2024-03-24T10:33:56.305754+0000","last_peered":"2024-03-24T10:33:56.305754+0000","last_clean":"2024-03-24T10:33:56.305754+0000","last_became_active":"2024-03-24T10:33:55.140774+0000","last_became_peered":"2024-03-24T10:33:55.140774+0000","last_unstale":"2024-03-24T10:33:56.305754+0000","last_undegraded":"2024-03-24T10:33:56.305754+0000","last_fullsized":"2024-03-24T10:33:56.305754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305846+0000","last_change":"2024-03-24T10:33:55.148019+0000","last_active":"2024-03-24T10:33:56.305846+0000","last_peered":"2024-03-24T10:33:56.305846+0000","last_clean":"2024-03-24T10:33:56.305846+0000","last_became_active":"2024-03-24T10:33:55.145313+0000","last_became_peered":"2024-03-24T10:33:55.145313+0000","last_unstale":"2024-03-24T10:33:56.305846+0000","last_undegraded":"2024-03-24T10:33:56.305846+0000","last_fullsized":"2024-03-24T10:33:56.305846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306869+0000","last_change":"2024-03-24T10:33:55.145334+0000","last_active":"2024-03-24T10:33:56.306869+0000","last_peered":"2024-03-24T10:33:56.306869+0000","last_clean":"2024-03-24T10:33:56.306869+0000","last_became_active":"2024-03-24T10:33:55.145018+0000","last_became_peered":"2024-03-24T10:33:55.145018+0000","last_unstale":"2024-03-24T10:33:56.306869+0000","last_undegraded":"2024-03-24T10:33:56.306869+0000","last_fullsized":"2024-03-24T10:33:56.306869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:24:24.292025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305901+0000","last_change":"2024-03-24T10:33:55.142392+0000","last_active":"2024-03-24T10:33:56.305901+0000","last_peered":"2024-03-24T10:33:56.305901+0000","last_clean":"2024-03-24T10:33:56.305901+0000","last_became_active":"2024-03-24T10:33:55.142070+0000","last_became_peered":"2024-03-24T10:33:55.142070+0000","last_unstale":"2024-03-24T10:33:56.305901+0000","last_undegraded":"2024-03-24T10:33:56.305901+0000","last_fullsized":"2024-03-24T10:33:56.305901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306206+0000","last_change":"2024-03-24T10:33:55.158958+0000","last_active":"2024-03-24T10:33:56.306206+0000","last_peered":"2024-03-24T10:33:56.306206+0000","last_clean":"2024-03-24T10:33:56.306206+0000","last_became_active":"2024-03-24T10:33:55.158270+0000","last_became_peered":"2024-03-24T10:33:55.158270+0000","last_unstale":"2024-03-24T10:33:56.306206+0000","last_undegraded":"2024-03-24T10:33:56.306206+0000","last_fullsized":"2024-03-24T10:33:56.306206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305233+0000","last_change":"2024-03-24T10:33:55.142337+0000","last_active":"2024-03-24T10:33:56.305233+0000","last_peered":"2024-03-24T10:33:56.305233+0000","last_clean":"2024-03-24T10:33:56.305233+0000","last_became_active":"2024-03-24T10:33:55.141994+0000","last_became_peered":"2024-03-24T10:33:55.141994+0000","last_unstale":"2024-03-24T10:33:56.305233+0000","last_undegraded":"2024-03-24T10:33:56.305233+0000","last_fullsized":"2024-03-24T10:33:56.305233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306802+0000","last_change":"2024-03-24T10:33:55.155886+0000","last_active":"2024-03-24T10:33:56.306802+0000","last_peered":"2024-03-24T10:33:56.306802+0000","last_clean":"2024-03-24T10:33:56.306802+0000","last_became_active":"2024-03-24T10:33:55.153792+0000","last_became_peered":"2024-03-24T10:33:55.153792+0000","last_unstale":"2024-03-24T10:33:56.306802+0000","last_undegraded":"2024-03-24T10:33:56.306802+0000","last_fullsized":"2024-03-24T10:33:56.306802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124307+0000","last_change":"2024-03-24T10:33:55.152791+0000","last_active":"2024-03-24T10:33:56.124307+0000","last_peered":"2024-03-24T10:33:56.124307+0000","last_clean":"2024-03-24T10:33:56.124307+0000","last_became_active":"2024-03-24T10:33:55.152524+0000","last_became_peered":"2024-03-24T10:33:55.152524+0000","last_unstale":"2024-03-24T10:33:56.124307+0000","last_undegraded":"2024-03-24T10:33:56.124307+0000","last_fullsized":"2024-03-24T10:33:56.124307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124236+0000","last_change":"2024-03-24T10:33:55.153894+0000","last_active":"2024-03-24T10:33:56.124236+0000","last_peered":"2024-03-24T10:33:56.124236+0000","last_clean":"2024-03-24T10:33:56.124236+0000","last_became_active":"2024-03-24T10:33:55.153623+0000","last_became_peered":"2024-03-24T10:33:55.153623+0000","last_unstale":"2024-03-24T10:33:56.124236+0000","last_undegraded":"2024-03-24T10:33:56.124236+0000","last_fullsized":"2024-03-24T10:33:56.124236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306325+0000","last_change":"2024-03-24T10:33:55.148337+0000","last_active":"2024-03-24T10:33:56.306325+0000","last_peered":"2024-03-24T10:33:56.306325+0000","last_clean":"2024-03-24T10:33:56.306325+0000","last_became_active":"2024-03-24T10:33:55.148076+0000","last_became_peered":"2024-03-24T10:33:55.148076+0000","last_unstale":"2024-03-24T10:33:56.306325+0000","last_undegraded":"2024-03-24T10:33:56.306325+0000","last_fullsized":"2024-03-24T10:33:56.306325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305500+0000","last_change":"2024-03-24T10:33:55.148625+0000","last_active":"2024-03-24T10:33:56.305500+0000","last_peered":"2024-03-24T10:33:56.305500+0000","last_clean":"2024-03-24T10:33:56.305500+0000","last_became_active":"2024-03-24T10:33:55.148234+0000","last_became_peered":"2024-03-24T10:33:55.148234+0000","last_unstale":"2024-03-24T10:33:56.305500+0000","last_undegraded":"2024-03-24T10:33:56.305500+0000","last_fullsized":"2024-03-24T10:33:56.305500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305594+0000","last_change":"2024-03-24T10:33:55.141825+0000","last_active":"2024-03-24T10:33:56.305594+0000","last_peered":"2024-03-24T10:33:56.305594+0000","last_clean":"2024-03-24T10:33:56.305594+0000","last_became_active":"2024-03-24T10:33:55.141380+0000","last_became_peered":"2024-03-24T10:33:55.141380+0000","last_unstale":"2024-03-24T10:33:56.305594+0000","last_undegraded":"2024-03-24T10:33:56.305594+0000","last_fullsized":"2024-03-24T10:33:56.305594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:51.274453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122775+0000","last_change":"2024-03-24T10:33:55.132926+0000","last_active":"2024-03-24T10:33:56.122775+0000","last_peered":"2024-03-24T10:33:56.122775+0000","last_clean":"2024-03-24T10:33:56.122775+0000","last_became_active":"2024-03-24T10:33:55.132614+0000","last_became_peered":"2024-03-24T10:33:55.132614+0000","last_unstale":"2024-03-24T10:33:56.122775+0000","last_undegraded":"2024-03-24T10:33:56.122775+0000","last_fullsized":"2024-03-24T10:33:56.122775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305340+0000","last_change":"2024-03-24T10:33:55.141440+0000","last_active":"2024-03-24T10:33:56.305340+0000","last_peered":"2024-03-24T10:33:56.305340+0000","last_clean":"2024-03-24T10:33:56.305340+0000","last_became_active":"2024-03-24T10:33:55.141096+0000","last_became_peered":"2024-03-24T10:33:55.141096+0000","last_unstale":"2024-03-24T10:33:56.305340+0000","last_undegraded":"2024-03-24T10:33:56.305340+0000","last_fullsized":"2024-03-24T10:33:56.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306420+0000","last_change":"2024-03-24T10:33:55.152651+0000","last_active":"2024-03-24T10:33:56.306420+0000","last_peered":"2024-03-24T10:33:56.306420+0000","last_clean":"2024-03-24T10:33:56.306420+0000","last_became_active":"2024-03-24T10:33:55.152386+0000","last_became_peered":"2024-03-24T10:33:55.152386+0000","last_unstale":"2024-03-24T10:33:56.306420+0000","last_undegraded":"2024-03-24T10:33:56.306420+0000","last_fullsized":"2024-03-24T10:33:56.306420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123835+0000","last_change":"2024-03-24T10:33:55.149184+0000","last_active":"2024-03-24T10:33:56.123835+0000","last_peered":"2024-03-24T10:33:56.123835+0000","last_clean":"2024-03-24T10:33:56.123835+0000","last_became_active":"2024-03-24T10:33:55.148828+0000","last_became_peered":"2024-03-24T10:33:55.148828+0000","last_unstale":"2024-03-24T10:33:56.123835+0000","last_undegraded":"2024-03-24T10:33:56.123835+0000","last_fullsized":"2024-03-24T10:33:56.123835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:33.564193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306947+0000","last_change":"2024-03-24T10:33:55.147090+0000","last_active":"2024-03-24T10:33:56.306947+0000","last_peered":"2024-03-24T10:33:56.306947+0000","last_clean":"2024-03-24T10:33:56.306947+0000","last_became_active":"2024-03-24T10:33:55.146820+0000","last_became_peered":"2024-03-24T10:33:55.146820+0000","last_unstale":"2024-03-24T10:33:56.306947+0000","last_undegraded":"2024-03-24T10:33:56.306947+0000","last_fullsized":"2024-03-24T10:33:56.306947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:12.251747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124127+0000","last_change":"2024-03-24T10:33:55.156434+0000","last_active":"2024-03-24T10:33:56.124127+0000","last_peered":"2024-03-24T10:33:56.124127+0000","last_clean":"2024-03-24T10:33:56.124127+0000","last_became_active":"2024-03-24T10:33:55.156176+0000","last_became_peered":"2024-03-24T10:33:55.156176+0000","last_unstale":"2024-03-24T10:33:56.124127+0000","last_undegraded":"2024-03-24T10:33:56.124127+0000","last_fullsized":"2024-03-24T10:33:56.124127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305688+0000","last_change":"2024-03-24T10:33:55.155986+0000","last_active":"2024-03-24T10:33:56.305688+0000","last_peered":"2024-03-24T10:33:56.305688+0000","last_clean":"2024-03-24T10:33:56.305688+0000","last_became_active":"2024-03-24T10:33:55.155699+0000","last_became_peered":"2024-03-24T10:33:55.155699+0000","last_unstale":"2024-03-24T10:33:56.305688+0000","last_undegraded":"2024-03-24T10:33:56.305688+0000","last_fullsized":"2024-03-24T10:33:56.305688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:41.606563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305535+0000","last_change":"2024-03-24T10:33:55.151865+0000","last_active":"2024-03-24T10:33:56.305535+0000","last_peered":"2024-03-24T10:33:56.305535+0000","last_clean":"2024-03-24T10:33:56.305535+0000","last_became_active":"2024-03-24T10:33:55.151592+0000","last_became_peered":"2024-03-24T10:33:55.151592+0000","last_unstale":"2024-03-24T10:33:56.305535+0000","last_undegraded":"2024-03-24T10:33:56.305535+0000","last_fullsized":"2024-03-24T10:33:56.305535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305309+0000","last_change":"2024-03-24T10:33:55.152238+0000","last_active":"2024-03-24T10:33:56.305309+0000","last_peered":"2024-03-24T10:33:56.305309+0000","last_clean":"2024-03-24T10:33:56.305309+0000","last_became_active":"2024-03-24T10:33:55.151966+0000","last_became_peered":"2024-03-24T10:33:55.151966+0000","last_unstale":"2024-03-24T10:33:56.305309+0000","last_undegraded":"2024-03-24T10:33:56.305309+0000","last_fullsized":"2024-03-24T10:33:56.305309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305757+0000","last_change":"2024-03-24T10:33:55.144173+0000","last_active":"2024-03-24T10:33:56.305757+0000","last_peered":"2024-03-24T10:33:56.305757+0000","last_clean":"2024-03-24T10:33:56.305757+0000","last_became_active":"2024-03-24T10:33:55.143874+0000","last_became_peered":"2024-03-24T10:33:55.143874+0000","last_unstale":"2024-03-24T10:33:56.305757+0000","last_undegraded":"2024-03-24T10:33:56.305757+0000","last_fullsized":"2024-03-24T10:33:56.305757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.136315+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.135994+0000","last_became_peered":"2024-03-24T10:33:55.135994+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124458+0000","last_change":"2024-03-24T10:33:55.159140+0000","last_active":"2024-03-24T10:33:56.124458+0000","last_peered":"2024-03-24T10:33:56.124458+0000","last_clean":"2024-03-24T10:33:56.124458+0000","last_became_active":"2024-03-24T10:33:55.158875+0000","last_became_peered":"2024-03-24T10:33:55.158875+0000","last_unstale":"2024-03-24T10:33:56.124458+0000","last_undegraded":"2024-03-24T10:33:56.124458+0000","last_fullsized":"2024-03-24T10:33:56.124458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123157+0000","last_change":"2024-03-24T10:33:55.134527+0000","last_active":"2024-03-24T10:33:56.123157+0000","last_peered":"2024-03-24T10:33:56.123157+0000","last_clean":"2024-03-24T10:33:56.123157+0000","last_became_active":"2024-03-24T10:33:55.134220+0000","last_became_peered":"2024-03-24T10:33:55.134220+0000","last_unstale":"2024-03-24T10:33:56.123157+0000","last_undegraded":"2024-03-24T10:33:56.123157+0000","last_fullsized":"2024-03-24T10:33:56.123157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306047+0000","last_change":"2024-03-24T10:33:55.138237+0000","last_active":"2024-03-24T10:33:56.306047+0000","last_peered":"2024-03-24T10:33:56.306047+0000","last_clean":"2024-03-24T10:33:56.306047+0000","last_became_active":"2024-03-24T10:33:55.136755+0000","last_became_peered":"2024-03-24T10:33:55.136755+0000","last_unstale":"2024-03-24T10:33:56.306047+0000","last_undegraded":"2024-03-24T10:33:56.306047+0000","last_fullsized":"2024-03-24T10:33:56.306047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305840+0000","last_change":"2024-03-24T10:33:55.145372+0000","last_active":"2024-03-24T10:33:56.305840+0000","last_peered":"2024-03-24T10:33:56.305840+0000","last_clean":"2024-03-24T10:33:56.305840+0000","last_became_active":"2024-03-24T10:33:55.144207+0000","last_became_peered":"2024-03-24T10:33:55.144207+0000","last_unstale":"2024-03-24T10:33:56.305840+0000","last_undegraded":"2024-03-24T10:33:56.305840+0000","last_fullsized":"2024-03-24T10:33:56.305840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306010+0000","last_change":"2024-03-24T10:33:55.146001+0000","last_active":"2024-03-24T10:33:56.306010+0000","last_peered":"2024-03-24T10:33:56.306010+0000","last_clean":"2024-03-24T10:33:56.306010+0000","last_became_active":"2024-03-24T10:33:55.145472+0000","last_became_peered":"2024-03-24T10:33:55.145472+0000","last_unstale":"2024-03-24T10:33:56.306010+0000","last_undegraded":"2024-03-24T10:33:56.306010+0000","last_fullsized":"2024-03-24T10:33:56.306010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123635+0000","last_change":"2024-03-24T10:33:55.143320+0000","last_active":"2024-03-24T10:33:56.123635+0000","last_peered":"2024-03-24T10:33:56.123635+0000","last_clean":"2024-03-24T10:33:56.123635+0000","last_became_active":"2024-03-24T10:33:55.143046+0000","last_became_peered":"2024-03-24T10:33:55.143046+0000","last_unstale":"2024-03-24T10:33:56.123635+0000","last_undegraded":"2024-03-24T10:33:56.123635+0000","last_fullsized":"2024-03-24T10:33:56.123635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124493+0000","last_change":"2024-03-24T10:33:55.147275+0000","last_active":"2024-03-24T10:33:56.124493+0000","last_peered":"2024-03-24T10:33:56.124493+0000","last_clean":"2024-03-24T10:33:56.124493+0000","last_became_active":"2024-03-24T10:33:55.146476+0000","last_became_peered":"2024-03-24T10:33:55.146476+0000","last_unstale":"2024-03-24T10:33:56.124493+0000","last_undegraded":"2024-03-24T10:33:56.124493+0000","last_fullsized":"2024-03-24T10:33:56.124493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305716+0000","last_change":"2024-03-24T10:33:55.131787+0000","last_active":"2024-03-24T10:33:56.305716+0000","last_peered":"2024-03-24T10:33:56.305716+0000","last_clean":"2024-03-24T10:33:56.305716+0000","last_became_active":"2024-03-24T10:33:55.131462+0000","last_became_peered":"2024-03-24T10:33:55.131462+0000","last_unstale":"2024-03-24T10:33:56.305716+0000","last_undegraded":"2024-03-24T10:33:56.305716+0000","last_fullsized":"2024-03-24T10:33:56.305716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:24.900927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305842+0000","last_change":"2024-03-24T10:33:54.130171+0000","last_active":"2024-03-24T10:33:56.305842+0000","last_peered":"2024-03-24T10:33:56.305842+0000","last_clean":"2024-03-24T10:33:56.305842+0000","last_became_active":"2024-03-24T10:33:54.129845+0000","last_became_peered":"2024-03-24T10:33:54.129845+0000","last_unstale":"2024-03-24T10:33:56.305842+0000","last_undegraded":"2024-03-24T10:33:56.305842+0000","last_fullsized":"2024-03-24T10:33:56.305842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305988+0000","last_change":"2024-03-24T10:33:55.156576+0000","last_active":"2024-03-24T10:33:56.305988+0000","last_peered":"2024-03-24T10:33:56.305988+0000","last_clean":"2024-03-24T10:33:56.305988+0000","last_became_active":"2024-03-24T10:33:55.156301+0000","last_became_peered":"2024-03-24T10:33:55.156301+0000","last_unstale":"2024-03-24T10:33:56.305988+0000","last_undegraded":"2024-03-24T10:33:56.305988+0000","last_fullsized":"2024-03-24T10:33:56.305988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124113+0000","last_change":"2024-03-24T10:33:54.122838+0000","last_active":"2024-03-24T10:33:56.124113+0000","last_peered":"2024-03-24T10:33:56.124113+0000","last_clean":"2024-03-24T10:33:56.124113+0000","last_became_active":"2024-03-24T10:33:54.122012+0000","last_became_peered":"2024-03-24T10:33:54.122012+0000","last_unstale":"2024-03-24T10:33:56.124113+0000","last_undegraded":"2024-03-24T10:33:56.124113+0000","last_fullsized":"2024-03-24T10:33:56.124113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305341+0000","last_change":"2024-03-24T10:33:54.127097+0000","last_active":"2024-03-24T10:33:56.305341+0000","last_peered":"2024-03-24T10:33:56.305341+0000","last_clean":"2024-03-24T10:33:56.305341+0000","last_became_active":"2024-03-24T10:33:54.126575+0000","last_became_peered":"2024-03-24T10:33:54.126575+0000","last_unstale":"2024-03-24T10:33:56.305341+0000","last_undegraded":"2024-03-24T10:33:56.305341+0000","last_fullsized":"2024-03-24T10:33:56.305341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305318+0000","last_change":"2024-03-24T10:33:55.138621+0000","last_active":"2024-03-24T10:33:56.305318+0000","last_peered":"2024-03-24T10:33:56.305318+0000","last_clean":"2024-03-24T10:33:56.305318+0000","last_became_active":"2024-03-24T10:33:55.138352+0000","last_became_peered":"2024-03-24T10:33:55.138352+0000","last_unstale":"2024-03-24T10:33:56.305318+0000","last_undegraded":"2024-03-24T10:33:56.305318+0000","last_fullsized":"2024-03-24T10:33:56.305318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305437+0000","last_change":"2024-03-24T10:33:54.125263+0000","last_active":"2024-03-24T10:33:56.305437+0000","last_peered":"2024-03-24T10:33:56.305437+0000","last_clean":"2024-03-24T10:33:56.305437+0000","last_became_active":"2024-03-24T10:33:54.124945+0000","last_became_peered":"2024-03-24T10:33:54.124945+0000","last_unstale":"2024-03-24T10:33:56.305437+0000","last_undegraded":"2024-03-24T10:33:56.305437+0000","last_fullsized":"2024-03-24T10:33:56.305437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122873+0000","last_change":"2024-03-24T10:33:55.134395+0000","last_active":"2024-03-24T10:33:56.122873+0000","last_peered":"2024-03-24T10:33:56.122873+0000","last_clean":"2024-03-24T10:33:56.122873+0000","last_became_active":"2024-03-24T10:33:55.133957+0000","last_became_peered":"2024-03-24T10:33:55.133957+0000","last_unstale":"2024-03-24T10:33:56.122873+0000","last_undegraded":"2024-03-24T10:33:56.122873+0000","last_fullsized":"2024-03-24T10:33:56.122873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124381+0000","last_change":"2024-03-24T10:33:54.127218+0000","last_active":"2024-03-24T10:33:56.124381+0000","last_peered":"2024-03-24T10:33:56.124381+0000","last_clean":"2024-03-24T10:33:56.124381+0000","last_became_active":"2024-03-24T10:33:54.126932+0000","last_became_peered":"2024-03-24T10:33:54.126932+0000","last_unstale":"2024-03-24T10:33:56.124381+0000","last_undegraded":"2024-03-24T10:33:56.124381+0000","last_fullsized":"2024-03-24T10:33:56.124381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306066+0000","last_change":"2024-03-24T10:33:55.150999+0000","last_active":"2024-03-24T10:33:56.306066+0000","last_peered":"2024-03-24T10:33:56.306066+0000","last_clean":"2024-03-24T10:33:56.306066+0000","last_became_active":"2024-03-24T10:33:55.150729+0000","last_became_peered":"2024-03-24T10:33:55.150729+0000","last_unstale":"2024-03-24T10:33:56.306066+0000","last_undegraded":"2024-03-24T10:33:56.306066+0000","last_fullsized":"2024-03-24T10:33:56.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306163+0000","last_change":"2024-03-24T10:33:54.126346+0000","last_active":"2024-03-24T10:33:56.306163+0000","last_peered":"2024-03-24T10:33:56.306163+0000","last_clean":"2024-03-24T10:33:56.306163+0000","last_became_active":"2024-03-24T10:33:54.126026+0000","last_became_peered":"2024-03-24T10:33:54.126026+0000","last_unstale":"2024-03-24T10:33:56.306163+0000","last_undegraded":"2024-03-24T10:33:56.306163+0000","last_fullsized":"2024-03-24T10:33:56.306163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306298+0000","last_change":"2024-03-24T10:33:55.145385+0000","last_active":"2024-03-24T10:33:56.306298+0000","last_peered":"2024-03-24T10:33:56.306298+0000","last_clean":"2024-03-24T10:33:56.306298+0000","last_became_active":"2024-03-24T10:33:55.144766+0000","last_became_peered":"2024-03-24T10:33:55.144766+0000","last_unstale":"2024-03-24T10:33:56.306298+0000","last_undegraded":"2024-03-24T10:33:56.306298+0000","last_fullsized":"2024-03-24T10:33:56.306298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123213+0000","last_change":"2024-03-24T10:33:54.128955+0000","last_active":"2024-03-24T10:33:56.123213+0000","last_peered":"2024-03-24T10:33:56.123213+0000","last_clean":"2024-03-24T10:33:56.123213+0000","last_became_active":"2024-03-24T10:33:54.128654+0000","last_became_peered":"2024-03-24T10:33:54.128654+0000","last_unstale":"2024-03-24T10:33:56.123213+0000","last_undegraded":"2024-03-24T10:33:56.123213+0000","last_fullsized":"2024-03-24T10:33:56.123213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306400+0000","last_change":"2024-03-24T10:33:55.146595+0000","last_active":"2024-03-24T10:33:56.306400+0000","last_peered":"2024-03-24T10:33:56.306400+0000","last_clean":"2024-03-24T10:33:56.306400+0000","last_became_active":"2024-03-24T10:33:55.146309+0000","last_became_peered":"2024-03-24T10:33:55.146309+0000","last_unstale":"2024-03-24T10:33:56.306400+0000","last_undegraded":"2024-03-24T10:33:56.306400+0000","last_fullsized":"2024-03-24T10:33:56.306400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124069+0000","last_change":"2024-03-24T10:33:54.125637+0000","last_active":"2024-03-24T10:33:56.124069+0000","last_peered":"2024-03-24T10:33:56.124069+0000","last_clean":"2024-03-24T10:33:56.124069+0000","last_became_active":"2024-03-24T10:33:54.125278+0000","last_became_peered":"2024-03-24T10:33:54.125278+0000","last_unstale":"2024-03-24T10:33:56.124069+0000","last_undegraded":"2024-03-24T10:33:56.124069+0000","last_fullsized":"2024-03-24T10:33:56.124069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306668+0000","last_change":"2024-03-24T10:33:55.156480+0000","last_active":"2024-03-24T10:33:56.306668+0000","last_peered":"2024-03-24T10:33:56.306668+0000","last_clean":"2024-03-24T10:33:56.306668+0000","last_became_active":"2024-03-24T10:33:55.156201+0000","last_became_peered":"2024-03-24T10:33:55.156201+0000","last_unstale":"2024-03-24T10:33:56.306668+0000","last_undegraded":"2024-03-24T10:33:56.306668+0000","last_fullsized":"2024-03-24T10:33:56.306668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306091+0000","last_change":"2024-03-24T10:33:54.125505+0000","last_active":"2024-03-24T10:33:56.306091+0000","last_peered":"2024-03-24T10:33:56.306091+0000","last_clean":"2024-03-24T10:33:56.306091+0000","last_became_active":"2024-03-24T10:33:54.125170+0000","last_became_peered":"2024-03-24T10:33:54.125170+0000","last_unstale":"2024-03-24T10:33:56.306091+0000","last_undegraded":"2024-03-24T10:33:56.306091+0000","last_fullsized":"2024-03-24T10:33:56.306091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305748+0000","last_change":"2024-03-24T10:33:55.156955+0000","last_active":"2024-03-24T10:33:56.305748+0000","last_peered":"2024-03-24T10:33:56.305748+0000","last_clean":"2024-03-24T10:33:56.305748+0000","last_became_active":"2024-03-24T10:33:55.156565+0000","last_became_peered":"2024-03-24T10:33:55.156565+0000","last_unstale":"2024-03-24T10:33:56.305748+0000","last_undegraded":"2024-03-24T10:33:56.305748+0000","last_fullsized":"2024-03-24T10:33:56.305748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:32.795374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306167+0000","last_change":"2024-03-24T10:33:54.128904+0000","last_active":"2024-03-24T10:33:56.306167+0000","last_peered":"2024-03-24T10:33:56.306167+0000","last_clean":"2024-03-24T10:33:56.306167+0000","last_became_active":"2024-03-24T10:33:54.128349+0000","last_became_peered":"2024-03-24T10:33:54.128349+0000","last_unstale":"2024-03-24T10:33:56.306167+0000","last_undegraded":"2024-03-24T10:33:56.306167+0000","last_fullsized":"2024-03-24T10:33:56.306167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306695+0000","last_change":"2024-03-24T10:33:55.155945+0000","last_active":"2024-03-24T10:33:56.306695+0000","last_peered":"2024-03-24T10:33:56.306695+0000","last_clean":"2024-03-24T10:33:56.306695+0000","last_became_active":"2024-03-24T10:33:55.155660+0000","last_became_peered":"2024-03-24T10:33:55.155660+0000","last_unstale":"2024-03-24T10:33:56.306695+0000","last_undegraded":"2024-03-24T10:33:56.306695+0000","last_fullsized":"2024-03-24T10:33:56.306695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124002+0000","last_change":"2024-03-24T10:33:54.127975+0000","last_active":"2024-03-24T10:33:56.124002+0000","last_peered":"2024-03-24T10:33:56.124002+0000","last_clean":"2024-03-24T10:33:56.124002+0000","last_became_active":"2024-03-24T10:33:54.127705+0000","last_became_peered":"2024-03-24T10:33:54.127705+0000","last_unstale":"2024-03-24T10:33:56.124002+0000","last_undegraded":"2024-03-24T10:33:56.124002+0000","last_fullsized":"2024-03-24T10:33:56.124002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124453+0000","last_change":"2024-03-24T10:33:55.146773+0000","last_active":"2024-03-24T10:33:56.124453+0000","last_peered":"2024-03-24T10:33:56.124453+0000","last_clean":"2024-03-24T10:33:56.124453+0000","last_became_active":"2024-03-24T10:33:55.146086+0000","last_became_peered":"2024-03-24T10:33:55.146086+0000","last_unstale":"2024-03-24T10:33:56.124453+0000","last_undegraded":"2024-03-24T10:33:56.124453+0000","last_fullsized":"2024-03-24T10:33:56.124453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123547+0000","last_change":"2024-03-24T10:33:54.114004+0000","last_active":"2024-03-24T10:33:56.123547+0000","last_peered":"2024-03-24T10:33:56.123547+0000","last_clean":"2024-03-24T10:33:56.123547+0000","last_became_active":"2024-03-24T10:33:54.113714+0000","last_became_peered":"2024-03-24T10:33:54.113714+0000","last_unstale":"2024-03-24T10:33:56.123547+0000","last_undegraded":"2024-03-24T10:33:56.123547+0000","last_fullsized":"2024-03-24T10:33:56.123547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306338+0000","last_change":"2024-03-24T10:33:55.160713+0000","last_active":"2024-03-24T10:33:56.306338+0000","last_peered":"2024-03-24T10:33:56.306338+0000","last_clean":"2024-03-24T10:33:56.306338+0000","last_became_active":"2024-03-24T10:33:55.157835+0000","last_became_peered":"2024-03-24T10:33:55.157835+0000","last_unstale":"2024-03-24T10:33:56.306338+0000","last_undegraded":"2024-03-24T10:33:56.306338+0000","last_fullsized":"2024-03-24T10:33:56.306338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306046+0000","last_change":"2024-03-24T10:33:54.123641+0000","last_active":"2024-03-24T10:33:56.306046+0000","last_peered":"2024-03-24T10:33:56.306046+0000","last_clean":"2024-03-24T10:33:56.306046+0000","last_became_active":"2024-03-24T10:33:54.123045+0000","last_became_peered":"2024-03-24T10:33:54.123045+0000","last_unstale":"2024-03-24T10:33:56.306046+0000","last_undegraded":"2024-03-24T10:33:56.306046+0000","last_fullsized":"2024-03-24T10:33:56.306046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:41:41.902140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306617+0000","last_change":"2024-03-24T10:33:55.155307+0000","last_active":"2024-03-24T10:33:56.306617+0000","last_peered":"2024-03-24T10:33:56.306617+0000","last_clean":"2024-03-24T10:33:56.306617+0000","last_became_active":"2024-03-24T10:33:55.154976+0000","last_became_peered":"2024-03-24T10:33:55.154976+0000","last_unstale":"2024-03-24T10:33:56.306617+0000","last_undegraded":"2024-03-24T10:33:56.306617+0000","last_fullsized":"2024-03-24T10:33:56.306617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123767+0000","last_change":"2024-03-24T10:33:54.124359+0000","last_active":"2024-03-24T10:33:56.123767+0000","last_peered":"2024-03-24T10:33:56.123767+0000","last_clean":"2024-03-24T10:33:56.123767+0000","last_became_active":"2024-03-24T10:33:54.124063+0000","last_became_peered":"2024-03-24T10:33:54.124063+0000","last_unstale":"2024-03-24T10:33:56.123767+0000","last_undegraded":"2024-03-24T10:33:56.123767+0000","last_fullsized":"2024-03-24T10:33:56.123767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306899+0000","last_change":"2024-03-24T10:33:55.159546+0000","last_active":"2024-03-24T10:33:56.306899+0000","last_peered":"2024-03-24T10:33:56.306899+0000","last_clean":"2024-03-24T10:33:56.306899+0000","last_became_active":"2024-03-24T10:33:55.159193+0000","last_became_peered":"2024-03-24T10:33:55.159193+0000","last_unstale":"2024-03-24T10:33:56.306899+0000","last_undegraded":"2024-03-24T10:33:56.306899+0000","last_fullsized":"2024-03-24T10:33:56.306899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:07.558259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123249+0000","last_change":"2024-03-24T10:33:54.127205+0000","last_active":"2024-03-24T10:33:56.123249+0000","last_peered":"2024-03-24T10:33:56.123249+0000","last_clean":"2024-03-24T10:33:56.123249+0000","last_became_active":"2024-03-24T10:33:54.126938+0000","last_became_peered":"2024-03-24T10:33:54.126938+0000","last_unstale":"2024-03-24T10:33:56.123249+0000","last_undegraded":"2024-03-24T10:33:56.123249+0000","last_fullsized":"2024-03-24T10:33:56.123249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305970+0000","last_change":"2024-03-24T10:33:55.150130+0000","last_active":"2024-03-24T10:33:56.305970+0000","last_peered":"2024-03-24T10:33:56.305970+0000","last_clean":"2024-03-24T10:33:56.305970+0000","last_became_active":"2024-03-24T10:33:55.145689+0000","last_became_peered":"2024-03-24T10:33:55.145689+0000","last_unstale":"2024-03-24T10:33:56.305970+0000","last_undegraded":"2024-03-24T10:33:56.305970+0000","last_fullsized":"2024-03-24T10:33:56.305970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123759+0000","last_change":"2024-03-24T10:33:54.131439+0000","last_active":"2024-03-24T10:33:56.123759+0000","last_peered":"2024-03-24T10:33:56.123759+0000","last_clean":"2024-03-24T10:33:56.123759+0000","last_became_active":"2024-03-24T10:33:54.131117+0000","last_became_peered":"2024-03-24T10:33:54.131117+0000","last_unstale":"2024-03-24T10:33:56.123759+0000","last_undegraded":"2024-03-24T10:33:56.123759+0000","last_fullsized":"2024-03-24T10:33:56.123759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306791+0000","last_change":"2024-03-24T10:33:55.157088+0000","last_active":"2024-03-24T10:33:56.306791+0000","last_peered":"2024-03-24T10:33:56.306791+0000","last_clean":"2024-03-24T10:33:56.306791+0000","last_became_active":"2024-03-24T10:33:55.156790+0000","last_became_peered":"2024-03-24T10:33:55.156790+0000","last_unstale":"2024-03-24T10:33:56.306791+0000","last_undegraded":"2024-03-24T10:33:56.306791+0000","last_fullsized":"2024-03-24T10:33:56.306791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306529+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:33:56.306529+0000","last_peered":"2024-03-24T10:33:56.306529+0000","last_clean":"2024-03-24T10:33:56.306529+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:33:56.306529+0000","last_undegraded":"2024-03-24T10:33:56.306529+0000","last_fullsized":"2024-03-24T10:33:56.306529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305276+0000","last_change":"2024-03-24T10:33:54.121820+0000","last_active":"2024-03-24T10:33:56.305276+0000","last_peered":"2024-03-24T10:33:56.305276+0000","last_clean":"2024-03-24T10:33:56.305276+0000","last_became_active":"2024-03-24T10:33:54.121496+0000","last_became_peered":"2024-03-24T10:33:54.121496+0000","last_unstale":"2024-03-24T10:33:56.305276+0000","last_undegraded":"2024-03-24T10:33:56.305276+0000","last_fullsized":"2024-03-24T10:33:56.305276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305400+0000","last_change":"2024-03-24T10:33:55.146810+0000","last_active":"2024-03-24T10:33:56.305400+0000","last_peered":"2024-03-24T10:33:56.305400+0000","last_clean":"2024-03-24T10:33:56.305400+0000","last_became_active":"2024-03-24T10:33:55.146508+0000","last_became_peered":"2024-03-24T10:33:55.146508+0000","last_unstale":"2024-03-24T10:33:56.305400+0000","last_undegraded":"2024-03-24T10:33:56.305400+0000","last_fullsized":"2024-03-24T10:33:56.305400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306540+0000","last_change":"2024-03-24T10:33:54.127727+0000","last_active":"2024-03-24T10:33:56.306540+0000","last_peered":"2024-03-24T10:33:56.306540+0000","last_clean":"2024-03-24T10:33:56.306540+0000","last_became_active":"2024-03-24T10:33:54.127106+0000","last_became_peered":"2024-03-24T10:33:54.127106+0000","last_unstale":"2024-03-24T10:33:56.306540+0000","last_undegraded":"2024-03-24T10:33:56.306540+0000","last_fullsized":"2024-03-24T10:33:56.306540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:31.791505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306196+0000","last_change":"2024-03-24T10:33:55.156425+0000","last_active":"2024-03-24T10:33:56.306196+0000","last_peered":"2024-03-24T10:33:56.306196+0000","last_clean":"2024-03-24T10:33:56.306196+0000","last_became_active":"2024-03-24T10:33:55.156146+0000","last_became_peered":"2024-03-24T10:33:55.156146+0000","last_unstale":"2024-03-24T10:33:56.306196+0000","last_undegraded":"2024-03-24T10:33:56.306196+0000","last_fullsized":"2024-03-24T10:33:56.306196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123643+0000","last_change":"2024-03-24T10:33:54.127887+0000","last_active":"2024-03-24T10:33:56.123643+0000","last_peered":"2024-03-24T10:33:56.123643+0000","last_clean":"2024-03-24T10:33:56.123643+0000","last_became_active":"2024-03-24T10:33:54.127602+0000","last_became_peered":"2024-03-24T10:33:54.127602+0000","last_unstale":"2024-03-24T10:33:56.123643+0000","last_undegraded":"2024-03-24T10:33:56.123643+0000","last_fullsized":"2024-03-24T10:33:56.123643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122824+0000","last_change":"2024-03-24T10:33:55.128811+0000","last_active":"2024-03-24T10:33:56.122824+0000","last_peered":"2024-03-24T10:33:56.122824+0000","last_clean":"2024-03-24T10:33:56.122824+0000","last_became_active":"2024-03-24T10:33:55.128094+0000","last_became_peered":"2024-03-24T10:33:55.128094+0000","last_unstale":"2024-03-24T10:33:56.122824+0000","last_undegraded":"2024-03-24T10:33:56.122824+0000","last_fullsized":"2024-03-24T10:33:56.122824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306104+0000","last_change":"2024-03-24T10:33:54.123812+0000","last_active":"2024-03-24T10:33:56.306104+0000","last_peered":"2024-03-24T10:33:56.306104+0000","last_clean":"2024-03-24T10:33:56.306104+0000","last_became_active":"2024-03-24T10:33:54.122614+0000","last_became_peered":"2024-03-24T10:33:54.122614+0000","last_unstale":"2024-03-24T10:33:56.306104+0000","last_undegraded":"2024-03-24T10:33:56.306104+0000","last_fullsized":"2024-03-24T10:33:56.306104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305946+0000","last_change":"2024-03-24T10:33:55.140461+0000","last_active":"2024-03-24T10:33:56.305946+0000","last_peered":"2024-03-24T10:33:56.305946+0000","last_clean":"2024-03-24T10:33:56.305946+0000","last_became_active":"2024-03-24T10:33:55.139737+0000","last_became_peered":"2024-03-24T10:33:55.139737+0000","last_unstale":"2024-03-24T10:33:56.305946+0000","last_undegraded":"2024-03-24T10:33:56.305946+0000","last_fullsized":"2024-03-24T10:33:56.305946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:27.228493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122944+0000","last_change":"2024-03-24T10:33:54.123506+0000","last_active":"2024-03-24T10:33:56.122944+0000","last_peered":"2024-03-24T10:33:56.122944+0000","last_clean":"2024-03-24T10:33:56.122944+0000","last_became_active":"2024-03-24T10:33:54.123194+0000","last_became_peered":"2024-03-24T10:33:54.123194+0000","last_unstale":"2024-03-24T10:33:56.122944+0000","last_undegraded":"2024-03-24T10:33:56.122944+0000","last_fullsized":"2024-03-24T10:33:56.122944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305900+0000","last_change":"2024-03-24T10:33:55.148696+0000","last_active":"2024-03-24T10:33:56.305900+0000","last_peered":"2024-03-24T10:33:56.305900+0000","last_clean":"2024-03-24T10:33:56.305900+0000","last_became_active":"2024-03-24T10:33:55.148382+0000","last_became_peered":"2024-03-24T10:33:55.148382+0000","last_unstale":"2024-03-24T10:33:56.305900+0000","last_undegraded":"2024-03-24T10:33:56.305900+0000","last_fullsized":"2024-03-24T10:33:56.305900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306970+0000","last_change":"2024-03-24T10:33:54.129296+0000","last_active":"2024-03-24T10:33:56.306970+0000","last_peered":"2024-03-24T10:33:56.306970+0000","last_clean":"2024-03-24T10:33:56.306970+0000","last_became_active":"2024-03-24T10:33:54.128788+0000","last_became_peered":"2024-03-24T10:33:54.128788+0000","last_unstale":"2024-03-24T10:33:56.306970+0000","last_undegraded":"2024-03-24T10:33:56.306970+0000","last_fullsized":"2024-03-24T10:33:56.306970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:48.412131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122832+0000","last_change":"2024-03-24T10:33:55.128461+0000","last_active":"2024-03-24T10:33:56.122832+0000","last_peered":"2024-03-24T10:33:56.122832+0000","last_clean":"2024-03-24T10:33:56.122832+0000","last_became_active":"2024-03-24T10:33:55.128125+0000","last_became_peered":"2024-03-24T10:33:55.128125+0000","last_unstale":"2024-03-24T10:33:56.122832+0000","last_undegraded":"2024-03-24T10:33:56.122832+0000","last_fullsized":"2024-03-24T10:33:56.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.140759+0000","last_change":"2024-03-24T10:33:54.117337+0000","last_active":"2024-03-24T10:33:56.140759+0000","last_peered":"2024-03-24T10:33:56.140759+0000","last_clean":"2024-03-24T10:33:56.140759+0000","last_became_active":"2024-03-24T10:33:54.116814+0000","last_became_peered":"2024-03-24T10:33:54.116814+0000","last_unstale":"2024-03-24T10:33:56.140759+0000","last_undegraded":"2024-03-24T10:33:56.140759+0000","last_fullsized":"2024-03-24T10:33:56.140759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.147551+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.147261+0000","last_became_peered":"2024-03-24T10:33:55.147261+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124432+0000","last_change":"2024-03-24T10:33:54.127434+0000","last_active":"2024-03-24T10:33:56.124432+0000","last_peered":"2024-03-24T10:33:56.124432+0000","last_clean":"2024-03-24T10:33:56.124432+0000","last_became_active":"2024-03-24T10:33:54.127162+0000","last_became_peered":"2024-03-24T10:33:54.127162+0000","last_unstale":"2024-03-24T10:33:56.124432+0000","last_undegraded":"2024-03-24T10:33:56.124432+0000","last_fullsized":"2024-03-24T10:33:56.124432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306039+0000","last_change":"2024-03-24T10:33:55.142227+0000","last_active":"2024-03-24T10:33:56.306039+0000","last_peered":"2024-03-24T10:33:56.306039+0000","last_clean":"2024-03-24T10:33:56.306039+0000","last_became_active":"2024-03-24T10:33:55.141920+0000","last_became_peered":"2024-03-24T10:33:55.141920+0000","last_unstale":"2024-03-24T10:33:56.306039+0000","last_undegraded":"2024-03-24T10:33:56.306039+0000","last_fullsized":"2024-03-24T10:33:56.306039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306452+0000","last_change":"2024-03-24T10:33:54.128286+0000","last_active":"2024-03-24T10:33:56.306452+0000","last_peered":"2024-03-24T10:33:56.306452+0000","last_clean":"2024-03-24T10:33:56.306452+0000","last_became_active":"2024-03-24T10:33:54.127434+0000","last_became_peered":"2024-03-24T10:33:54.127434+0000","last_unstale":"2024-03-24T10:33:56.306452+0000","last_undegraded":"2024-03-24T10:33:56.306452+0000","last_fullsized":"2024-03-24T10:33:56.306452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:40.602694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305662+0000","last_change":"2024-03-24T10:33:55.132582+0000","last_active":"2024-03-24T10:33:56.305662+0000","last_peered":"2024-03-24T10:33:56.305662+0000","last_clean":"2024-03-24T10:33:56.305662+0000","last_became_active":"2024-03-24T10:33:55.132259+0000","last_became_peered":"2024-03-24T10:33:55.132259+0000","last_unstale":"2024-03-24T10:33:56.305662+0000","last_undegraded":"2024-03-24T10:33:56.305662+0000","last_fullsized":"2024-03-24T10:33:56.305662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:22.596121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122894+0000","last_change":"2024-03-24T10:33:54.127899+0000","last_active":"2024-03-24T10:33:56.122894+0000","last_peered":"2024-03-24T10:33:56.122894+0000","last_clean":"2024-03-24T10:33:56.122894+0000","last_became_active":"2024-03-24T10:33:54.127592+0000","last_became_peered":"2024-03-24T10:33:54.127592+0000","last_unstale":"2024-03-24T10:33:56.122894+0000","last_undegraded":"2024-03-24T10:33:56.122894+0000","last_fullsized":"2024-03-24T10:33:56.122894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305592+0000","last_change":"2024-03-24T10:33:55.133853+0000","last_active":"2024-03-24T10:33:56.305592+0000","last_peered":"2024-03-24T10:33:56.305592+0000","last_clean":"2024-03-24T10:33:56.305592+0000","last_became_active":"2024-03-24T10:33:55.133478+0000","last_became_peered":"2024-03-24T10:33:55.133478+0000","last_unstale":"2024-03-24T10:33:56.305592+0000","last_undegraded":"2024-03-24T10:33:56.305592+0000","last_fullsized":"2024-03-24T10:33:56.305592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123001+0000","last_change":"2024-03-24T10:33:54.126665+0000","last_active":"2024-03-24T10:33:56.123001+0000","last_peered":"2024-03-24T10:33:56.123001+0000","last_clean":"2024-03-24T10:33:56.123001+0000","last_became_active":"2024-03-24T10:33:54.126339+0000","last_became_peered":"2024-03-24T10:33:54.126339+0000","last_unstale":"2024-03-24T10:33:56.123001+0000","last_undegraded":"2024-03-24T10:33:56.123001+0000","last_fullsized":"2024-03-24T10:33:56.123001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305669+0000","last_change":"2024-03-24T10:33:55.140585+0000","last_active":"2024-03-24T10:33:56.305669+0000","last_peered":"2024-03-24T10:33:56.305669+0000","last_clean":"2024-03-24T10:33:56.305669+0000","last_became_active":"2024-03-24T10:33:55.140142+0000","last_became_peered":"2024-03-24T10:33:55.140142+0000","last_unstale":"2024-03-24T10:33:56.305669+0000","last_undegraded":"2024-03-24T10:33:56.305669+0000","last_fullsized":"2024-03-24T10:33:56.305669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306757+0000","last_change":"2024-03-24T10:33:54.128102+0000","last_active":"2024-03-24T10:33:56.306757+0000","last_peered":"2024-03-24T10:33:56.306757+0000","last_clean":"2024-03-24T10:33:56.306757+0000","last_became_active":"2024-03-24T10:33:54.127077+0000","last_became_peered":"2024-03-24T10:33:54.127077+0000","last_unstale":"2024-03-24T10:33:56.306757+0000","last_undegraded":"2024-03-24T10:33:56.306757+0000","last_fullsized":"2024-03-24T10:33:56.306757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123921+0000","last_change":"2024-03-24T10:33:55.155558+0000","last_active":"2024-03-24T10:33:56.123921+0000","last_peered":"2024-03-24T10:33:56.123921+0000","last_clean":"2024-03-24T10:33:56.123921+0000","last_became_active":"2024-03-24T10:33:55.155291+0000","last_became_peered":"2024-03-24T10:33:55.155291+0000","last_unstale":"2024-03-24T10:33:56.123921+0000","last_undegraded":"2024-03-24T10:33:56.123921+0000","last_fullsized":"2024-03-24T10:33:56.123921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305743+0000","last_change":"2024-03-24T10:33:55.159176+0000","last_active":"2024-03-24T10:33:56.305743+0000","last_peered":"2024-03-24T10:33:56.305743+0000","last_clean":"2024-03-24T10:33:56.305743+0000","last_became_active":"2024-03-24T10:33:55.158885+0000","last_became_peered":"2024-03-24T10:33:55.158885+0000","last_unstale":"2024-03-24T10:33:56.305743+0000","last_undegraded":"2024-03-24T10:33:56.305743+0000","last_fullsized":"2024-03-24T10:33:56.305743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306353+0000","last_change":"2024-03-24T10:33:54.132298+0000","last_active":"2024-03-24T10:33:56.306353+0000","last_peered":"2024-03-24T10:33:56.306353+0000","last_clean":"2024-03-24T10:33:56.306353+0000","last_became_active":"2024-03-24T10:33:54.132009+0000","last_became_peered":"2024-03-24T10:33:54.132009+0000","last_unstale":"2024-03-24T10:33:56.306353+0000","last_undegraded":"2024-03-24T10:33:56.306353+0000","last_fullsized":"2024-03-24T10:33:56.306353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305825+0000","last_change":"2024-03-24T10:33:55.159162+0000","last_active":"2024-03-24T10:33:56.305825+0000","last_peered":"2024-03-24T10:33:56.305825+0000","last_clean":"2024-03-24T10:33:56.305825+0000","last_became_active":"2024-03-24T10:33:55.158898+0000","last_became_peered":"2024-03-24T10:33:55.158898+0000","last_unstale":"2024-03-24T10:33:56.305825+0000","last_undegraded":"2024-03-24T10:33:56.305825+0000","last_fullsized":"2024-03-24T10:33:56.305825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306863+0000","last_change":"2024-03-24T10:33:54.129068+0000","last_active":"2024-03-24T10:33:56.306863+0000","last_peered":"2024-03-24T10:33:56.306863+0000","last_clean":"2024-03-24T10:33:56.306863+0000","last_became_active":"2024-03-24T10:33:54.128465+0000","last_became_peered":"2024-03-24T10:33:54.128465+0000","last_unstale":"2024-03-24T10:33:56.306863+0000","last_undegraded":"2024-03-24T10:33:56.306863+0000","last_fullsized":"2024-03-24T10:33:56.306863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306248+0000","last_change":"2024-03-24T10:33:55.146210+0000","last_active":"2024-03-24T10:33:56.306248+0000","last_peered":"2024-03-24T10:33:56.306248+0000","last_clean":"2024-03-24T10:33:56.306248+0000","last_became_active":"2024-03-24T10:33:55.145069+0000","last_became_peered":"2024-03-24T10:33:55.145069+0000","last_unstale":"2024-03-24T10:33:56.306248+0000","last_undegraded":"2024-03-24T10:33:56.306248+0000","last_fullsized":"2024-03-24T10:33:56.306248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305927+0000","last_change":"2024-03-24T10:33:54.126679+0000","last_active":"2024-03-24T10:33:56.305927+0000","last_peered":"2024-03-24T10:33:56.305927+0000","last_clean":"2024-03-24T10:33:56.305927+0000","last_became_active":"2024-03-24T10:33:54.126407+0000","last_became_peered":"2024-03-24T10:33:54.126407+0000","last_unstale":"2024-03-24T10:33:56.305927+0000","last_undegraded":"2024-03-24T10:33:56.305927+0000","last_fullsized":"2024-03-24T10:33:56.305927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305615+0000","last_change":"2024-03-24T10:33:55.150245+0000","last_active":"2024-03-24T10:33:56.305615+0000","last_peered":"2024-03-24T10:33:56.305615+0000","last_clean":"2024-03-24T10:33:56.305615+0000","last_became_active":"2024-03-24T10:33:55.149927+0000","last_became_peered":"2024-03-24T10:33:55.149927+0000","last_unstale":"2024-03-24T10:33:56.305615+0000","last_undegraded":"2024-03-24T10:33:56.305615+0000","last_fullsized":"2024-03-24T10:33:56.305615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306337+0000","last_change":"2024-03-24T10:33:54.127624+0000","last_active":"2024-03-24T10:33:56.306337+0000","last_peered":"2024-03-24T10:33:56.306337+0000","last_clean":"2024-03-24T10:33:56.306337+0000","last_became_active":"2024-03-24T10:33:54.127307+0000","last_became_peered":"2024-03-24T10:33:54.127307+0000","last_unstale":"2024-03-24T10:33:56.306337+0000","last_undegraded":"2024-03-24T10:33:56.306337+0000","last_fullsized":"2024-03-24T10:33:56.306337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:34:05.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:34:05.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.161+0000 7f76c1e8f640 1 Processor -- start 2024-03-24T10:34:05.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.161+0000 7f76c1e8f640 1 -- start start 2024-03-24T10:34:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76c1e8f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 0x7f76bc0d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76c1e8f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 0x7f76bc0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76c1e8f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76bc0d4eb0 con 0x7f76bc15e0a0 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76c1e8f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f76bc0d0160 con 0x7f76bc0cebc0 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76c1e8f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc0cf0e0 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 0x7f76bc0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76baffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 0x7f76bc0d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 0x7f76bc0b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33850/0 (socket says 172.21.15.1:33850) 2024-03-24T10:34:05.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76bbfff640 1 -- 172.21.15.1:0/1079253326 learned_addr learned my addr 172.21.15.1:0/1079253326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76bb7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f76bc15e0a0 0x7f76bc0d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55456/0 (socket says 172.21.15.1:55456) 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76baffd640 1 -- 172.21.15.1:0/1079253326 >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 msgr2=0x7f76bc0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76baffd640 1 --2- 172.21.15.1:0/1079253326 >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 0x7f76bc0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76baffd640 1 -- 172.21.15.1:0/1079253326 >> v1:172.21.15.3:6789/0 conn(0x7f76bc15e0a0 legacy=0x7f76bc0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76baffd640 1 -- 172.21.15.1:0/1079253326 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc0cf3c0 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.162+0000 7f76ba7fc640 1 -- 172.21.15.1:0/1079253326 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979857302 0 0) 0x7f76ac001110 con 0x7f76bc15e0a0 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.163+0000 7f76baffd640 1 --2- 172.21.15.1:0/1079253326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 0x7f76bc0d4970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76b00097a0 tx=0x7f76b002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2111a8d918827ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.163+0000 7f76ba7fc640 1 -- 172.21.15.1:0/1079253326 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0030610 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.163+0000 7f76ba7fc640 1 -- 172.21.15.1:0/1079253326 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b0030770 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.163+0000 7f76ba7fc640 1 -- 172.21.15.1:0/1079253326 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0030b00 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 -- 172.21.15.1:0/1079253326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 msgr2=0x7f76bc0d4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/1079253326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 0x7f76bc0d4970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76b00097a0 tx=0x7f76b002b960 comp rx=0 tx=0).stop 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 -- 172.21.15.1:0/1079253326 shutdown_connections 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/1079253326 >> v2:172.21.15.1:3300/0 conn(0x7f76bc0cebc0 0x7f76bc0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/1079253326 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0d2ca0 0x7f76bc0d4970 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 -- 172.21.15.1:0/1079253326 >> 172.21.15.1:0/1079253326 conn(0x7f76bc09d1c0 msgr2=0x7f76bc0d3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 -- 172.21.15.1:0/1079253326 shutdown_connections 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.164+0000 7f76c1e8f640 1 -- 172.21.15.1:0/1079253326 wait complete. 2024-03-24T10:34:05.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.165+0000 7f76c1e8f640 1 Processor -- start 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.165+0000 7f76c1e8f640 1 -- start start 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0cebc0 0x7f76bc0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76bc0d2ca0 0x7f76bc0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc0ce590 con 0x7f76bc15e0a0 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc0ca7a0 con 0x7f76bc0d2ca0 2024-03-24T10:34:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76c1e8f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc0b56c0 con 0x7f76bc0cebc0 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33856/0 (socket says 172.21.15.1:33856) 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 -- 172.21.15.1:0/3459769217 learned_addr learned my addr 172.21.15.1:0/3459769217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bb7fe640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0cebc0 0x7f76bc0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76baffd640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76bc0d2ca0 0x7f76bc0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 -- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0cebc0 msgr2=0x7f76bc0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0cebc0 0x7f76bc0cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 -- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76bc0d2ca0 msgr2=0x7f76bc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76bc0d2ca0 0x7f76bc0ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 -- 172.21.15.1:0/3459769217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b0009400 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76bbfff640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009d60 tx=0x7f76b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.166+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b400cd20 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.167+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b400f040 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.167+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4003bd0 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.167+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc0b4dd0 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.167+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc0b5070 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.168+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76bc0d4120 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.168+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f76b4003d30 con 0x7f76bc15e0a0 2024-03-24T10:34:05.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.168+0000 7f76a3fff640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 0x7f76a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.168+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f76b408a640 con 0x7f76bc15e0a0 2024-03-24T10:34:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.169+0000 7f76bb7fe640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 0x7f76a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.169+0000 7f76bb7fe640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 0x7f76a806dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f76ac007fd0 tx=0x7f76ac007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.174+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f76b40551a0 con 0x7f76bc15e0a0 2024-03-24T10:34:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.352+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76bc0cfc60 con 0x7f76a806b810 2024-03-24T10:34:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.358+0000 7f76a3fff640 1 -- 172.21.15.1:0/3459769217 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f76ac004c70 con 0x7f76a806b810 2024-03-24T10:34:05.364 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:05.366 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:34:05.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 msgr2=0x7f76a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 0x7f76a806dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f76ac007fd0 tx=0x7f76ac007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 msgr2=0x7f76bc0b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009d60 tx=0x7f76b400c490 comp rx=0 tx=0).stop 2024-03-24T10:34:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 shutdown_connections 2024-03-24T10:34:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f76a806b810 0x7f76a806dcd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76bc15e0a0 0x7f76bc0b9880 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76bc0d2ca0 0x7f76bc0ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 --2- 172.21.15.1:0/3459769217 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f76bc0cebc0 0x7f76bc0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 >> 172.21.15.1:0/3459769217 conn(0x7f76bc09d1c0 msgr2=0x7f76bc15f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 shutdown_connections 2024-03-24T10:34:05.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.361+0000 7f76c1e8f640 1 -- 172.21.15.1:0/3459769217 wait complete. 2024-03-24T10:34:05.373 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-24T10:34:04.519712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235752,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738968,"statfs":{"total":773094113280,"available":772852703232,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.399262"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306747+0000","last_change":"2024-03-24T10:33:55.160859+0000","last_active":"2024-03-24T10:33:56.306747+0000","last_peered":"2024-03-24T10:33:56.306747+0000","last_clean":"2024-03-24T10:33:56.306747+0000","last_became_active":"2024-03-24T10:33:55.160069+0000","last_became_peered":"2024-03-24T10:33:55.160069+0000","last_unstale":"2024-03-24T10:33:56.306747+0000","last_undegraded":"2024-03-24T10:33:56.306747+0000","last_fullsized":"2024-03-24T10:33:56.306747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123102+0000","last_change":"2024-03-24T10:33:55.140972+0000","last_active":"2024-03-24T10:33:56.123102+0000","last_peered":"2024-03-24T10:33:56.123102+0000","last_clean":"2024-03-24T10:33:56.123102+0000","last_became_active":"2024-03-24T10:33:55.140573+0000","last_became_peered":"2024-03-24T10:33:55.140573+0000","last_unstale":"2024-03-24T10:33:56.123102+0000","last_undegraded":"2024-03-24T10:33:56.123102+0000","last_fullsized":"2024-03-24T10:33:56.123102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305754+0000","last_change":"2024-03-24T10:33:55.141078+0000","last_active":"2024-03-24T10:33:56.305754+0000","last_peered":"2024-03-24T10:33:56.305754+0000","last_clean":"2024-03-24T10:33:56.305754+0000","last_became_active":"2024-03-24T10:33:55.140774+0000","last_became_peered":"2024-03-24T10:33:55.140774+0000","last_unstale":"2024-03-24T10:33:56.305754+0000","last_undegraded":"2024-03-24T10:33:56.305754+0000","last_fullsized":"2024-03-24T10:33:56.305754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305846+0000","last_change":"2024-03-24T10:33:55.148019+0000","last_active":"2024-03-24T10:33:56.305846+0000","last_peered":"2024-03-24T10:33:56.305846+0000","last_clean":"2024-03-24T10:33:56.305846+0000","last_became_active":"2024-03-24T10:33:55.145313+0000","last_became_peered":"2024-03-24T10:33:55.145313+0000","last_unstale":"2024-03-24T10:33:56.305846+0000","last_undegraded":"2024-03-24T10:33:56.305846+0000","last_fullsized":"2024-03-24T10:33:56.305846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306869+0000","last_change":"2024-03-24T10:33:55.145334+0000","last_active":"2024-03-24T10:33:56.306869+0000","last_peered":"2024-03-24T10:33:56.306869+0000","last_clean":"2024-03-24T10:33:56.306869+0000","last_became_active":"2024-03-24T10:33:55.145018+0000","last_became_peered":"2024-03-24T10:33:55.145018+0000","last_unstale":"2024-03-24T10:33:56.306869+0000","last_undegraded":"2024-03-24T10:33:56.306869+0000","last_fullsized":"2024-03-24T10:33:56.306869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:24:24.292025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305901+0000","last_change":"2024-03-24T10:33:55.142392+0000","last_active":"2024-03-24T10:33:56.305901+0000","last_peered":"2024-03-24T10:33:56.305901+0000","last_clean":"2024-03-24T10:33:56.305901+0000","last_became_active":"2024-03-24T10:33:55.142070+0000","last_became_peered":"2024-03-24T10:33:55.142070+0000","last_unstale":"2024-03-24T10:33:56.305901+0000","last_undegraded":"2024-03-24T10:33:56.305901+0000","last_fullsized":"2024-03-24T10:33:56.305901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306206+0000","last_change":"2024-03-24T10:33:55.158958+0000","last_active":"2024-03-24T10:33:56.306206+0000","last_peered":"2024-03-24T10:33:56.306206+0000","last_clean":"2024-03-24T10:33:56.306206+0000","last_became_active":"2024-03-24T10:33:55.158270+0000","last_became_peered":"2024-03-24T10:33:55.158270+0000","last_unstale":"2024-03-24T10:33:56.306206+0000","last_undegraded":"2024-03-24T10:33:56.306206+0000","last_fullsized":"2024-03-24T10:33:56.306206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305233+0000","last_change":"2024-03-24T10:33:55.142337+0000","last_active":"2024-03-24T10:33:56.305233+0000","last_peered":"2024-03-24T10:33:56.305233+0000","last_clean":"2024-03-24T10:33:56.305233+0000","last_became_active":"2024-03-24T10:33:55.141994+0000","last_became_peered":"2024-03-24T10:33:55.141994+0000","last_unstale":"2024-03-24T10:33:56.305233+0000","last_undegraded":"2024-03-24T10:33:56.305233+0000","last_fullsized":"2024-03-24T10:33:56.305233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306802+0000","last_change":"2024-03-24T10:33:55.155886+0000","last_active":"2024-03-24T10:33:56.306802+0000","last_peered":"2024-03-24T10:33:56.306802+0000","last_clean":"2024-03-24T10:33:56.306802+0000","last_became_active":"2024-03-24T10:33:55.153792+0000","last_became_peered":"2024-03-24T10:33:55.153792+0000","last_unstale":"2024-03-24T10:33:56.306802+0000","last_undegraded":"2024-03-24T10:33:56.306802+0000","last_fullsized":"2024-03-24T10:33:56.306802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124307+0000","last_change":"2024-03-24T10:33:55.152791+0000","last_active":"2024-03-24T10:33:56.124307+0000","last_peered":"2024-03-24T10:33:56.124307+0000","last_clean":"2024-03-24T10:33:56.124307+0000","last_became_active":"2024-03-24T10:33:55.152524+0000","last_became_peered":"2024-03-24T10:33:55.152524+0000","last_unstale":"2024-03-24T10:33:56.124307+0000","last_undegraded":"2024-03-24T10:33:56.124307+0000","last_fullsized":"2024-03-24T10:33:56.124307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124236+0000","last_change":"2024-03-24T10:33:55.153894+0000","last_active":"2024-03-24T10:33:56.124236+0000","last_peered":"2024-03-24T10:33:56.124236+0000","last_clean":"2024-03-24T10:33:56.124236+0000","last_became_active":"2024-03-24T10:33:55.153623+0000","last_became_peered":"2024-03-24T10:33:55.153623+0000","last_unstale":"2024-03-24T10:33:56.124236+0000","last_undegraded":"2024-03-24T10:33:56.124236+0000","last_fullsized":"2024-03-24T10:33:56.124236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306325+0000","last_change":"2024-03-24T10:33:55.148337+0000","last_active":"2024-03-24T10:33:56.306325+0000","last_peered":"2024-03-24T10:33:56.306325+0000","last_clean":"2024-03-24T10:33:56.306325+0000","last_became_active":"2024-03-24T10:33:55.148076+0000","last_became_peered":"2024-03-24T10:33:55.148076+0000","last_unstale":"2024-03-24T10:33:56.306325+0000","last_undegraded":"2024-03-24T10:33:56.306325+0000","last_fullsized":"2024-03-24T10:33:56.306325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305500+0000","last_change":"2024-03-24T10:33:55.148625+0000","last_active":"2024-03-24T10:33:56.305500+0000","last_peered":"2024-03-24T10:33:56.305500+0000","last_clean":"2024-03-24T10:33:56.305500+0000","last_became_active":"2024-03-24T10:33:55.148234+0000","last_became_peered":"2024-03-24T10:33:55.148234+0000","last_unstale":"2024-03-24T10:33:56.305500+0000","last_undegraded":"2024-03-24T10:33:56.305500+0000","last_fullsized":"2024-03-24T10:33:56.305500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305594+0000","last_change":"2024-03-24T10:33:55.141825+0000","last_active":"2024-03-24T10:33:56.305594+0000","last_peered":"2024-03-24T10:33:56.305594+0000","last_clean":"2024-03-24T10:33:56.305594+0000","last_became_active":"2024-03-24T10:33:55.141380+0000","last_became_peered":"2024-03-24T10:33:55.141380+0000","last_unstale":"2024-03-24T10:33:56.305594+0000","last_undegraded":"2024-03-24T10:33:56.305594+0000","last_fullsized":"2024-03-24T10:33:56.305594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:51.274453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122775+0000","last_change":"2024-03-24T10:33:55.132926+0000","last_active":"2024-03-24T10:33:56.122775+0000","last_peered":"2024-03-24T10:33:56.122775+0000","last_clean":"2024-03-24T10:33:56.122775+0000","last_became_active":"2024-03-24T10:33:55.132614+0000","last_became_peered":"2024-03-24T10:33:55.132614+0000","last_unstale":"2024-03-24T10:33:56.122775+0000","last_undegraded":"2024-03-24T10:33:56.122775+0000","last_fullsized":"2024-03-24T10:33:56.122775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305340+0000","last_change":"2024-03-24T10:33:55.141440+0000","last_active":"2024-03-24T10:33:56.305340+0000","last_peered":"2024-03-24T10:33:56.305340+0000","last_clean":"2024-03-24T10:33:56.305340+0000","last_became_active":"2024-03-24T10:33:55.141096+0000","last_became_peered":"2024-03-24T10:33:55.141096+0000","last_unstale":"2024-03-24T10:33:56.305340+0000","last_undegraded":"2024-03-24T10:33:56.305340+0000","last_fullsized":"2024-03-24T10:33:56.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306420+0000","last_change":"2024-03-24T10:33:55.152651+0000","last_active":"2024-03-24T10:33:56.306420+0000","last_peered":"2024-03-24T10:33:56.306420+0000","last_clean":"2024-03-24T10:33:56.306420+0000","last_became_active":"2024-03-24T10:33:55.152386+0000","last_became_peered":"2024-03-24T10:33:55.152386+0000","last_unstale":"2024-03-24T10:33:56.306420+0000","last_undegraded":"2024-03-24T10:33:56.306420+0000","last_fullsized":"2024-03-24T10:33:56.306420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:54:43.760332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123835+0000","last_change":"2024-03-24T10:33:55.149184+0000","last_active":"2024-03-24T10:33:56.123835+0000","last_peered":"2024-03-24T10:33:56.123835+0000","last_clean":"2024-03-24T10:33:56.123835+0000","last_became_active":"2024-03-24T10:33:55.148828+0000","last_became_peered":"2024-03-24T10:33:55.148828+0000","last_unstale":"2024-03-24T10:33:56.123835+0000","last_undegraded":"2024-03-24T10:33:56.123835+0000","last_fullsized":"2024-03-24T10:33:56.123835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:33.564193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306947+0000","last_change":"2024-03-24T10:33:55.147090+0000","last_active":"2024-03-24T10:33:56.306947+0000","last_peered":"2024-03-24T10:33:56.306947+0000","last_clean":"2024-03-24T10:33:56.306947+0000","last_became_active":"2024-03-24T10:33:55.146820+0000","last_became_peered":"2024-03-24T10:33:55.146820+0000","last_unstale":"2024-03-24T10:33:56.306947+0000","last_undegraded":"2024-03-24T10:33:56.306947+0000","last_fullsized":"2024-03-24T10:33:56.306947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:12.251747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124127+0000","last_change":"2024-03-24T10:33:55.156434+0000","last_active":"2024-03-24T10:33:56.124127+0000","last_peered":"2024-03-24T10:33:56.124127+0000","last_clean":"2024-03-24T10:33:56.124127+0000","last_became_active":"2024-03-24T10:33:55.156176+0000","last_became_peered":"2024-03-24T10:33:55.156176+0000","last_unstale":"2024-03-24T10:33:56.124127+0000","last_undegraded":"2024-03-24T10:33:56.124127+0000","last_fullsized":"2024-03-24T10:33:56.124127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305688+0000","last_change":"2024-03-24T10:33:55.155986+0000","last_active":"2024-03-24T10:33:56.305688+0000","last_peered":"2024-03-24T10:33:56.305688+0000","last_clean":"2024-03-24T10:33:56.305688+0000","last_became_active":"2024-03-24T10:33:55.155699+0000","last_became_peered":"2024-03-24T10:33:55.155699+0000","last_unstale":"2024-03-24T10:33:56.305688+0000","last_undegraded":"2024-03-24T10:33:56.305688+0000","last_fullsized":"2024-03-24T10:33:56.305688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:41.606563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305535+0000","last_change":"2024-03-24T10:33:55.151865+0000","last_active":"2024-03-24T10:33:56.305535+0000","last_peered":"2024-03-24T10:33:56.305535+0000","last_clean":"2024-03-24T10:33:56.305535+0000","last_became_active":"2024-03-24T10:33:55.151592+0000","last_became_peered":"2024-03-24T10:33:55.151592+0000","last_unstale":"2024-03-24T10:33:56.305535+0000","last_undegraded":"2024-03-24T10:33:56.305535+0000","last_fullsized":"2024-03-24T10:33:56.305535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305309+0000","last_change":"2024-03-24T10:33:55.152238+0000","last_active":"2024-03-24T10:33:56.305309+0000","last_peered":"2024-03-24T10:33:56.305309+0000","last_clean":"2024-03-24T10:33:56.305309+0000","last_became_active":"2024-03-24T10:33:55.151966+0000","last_became_peered":"2024-03-24T10:33:55.151966+0000","last_unstale":"2024-03-24T10:33:56.305309+0000","last_undegraded":"2024-03-24T10:33:56.305309+0000","last_fullsized":"2024-03-24T10:33:56.305309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305757+0000","last_change":"2024-03-24T10:33:55.144173+0000","last_active":"2024-03-24T10:33:56.305757+0000","last_peered":"2024-03-24T10:33:56.305757+0000","last_clean":"2024-03-24T10:33:56.305757+0000","last_became_active":"2024-03-24T10:33:55.143874+0000","last_became_peered":"2024-03-24T10:33:55.143874+0000","last_unstale":"2024-03-24T10:33:56.305757+0000","last_undegraded":"2024-03-24T10:33:56.305757+0000","last_fullsized":"2024-03-24T10:33:56.305757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.136315+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.135994+0000","last_became_peered":"2024-03-24T10:33:55.135994+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124458+0000","last_change":"2024-03-24T10:33:55.159140+0000","last_active":"2024-03-24T10:33:56.124458+0000","last_peered":"2024-03-24T10:33:56.124458+0000","last_clean":"2024-03-24T10:33:56.124458+0000","last_became_active":"2024-03-24T10:33:55.158875+0000","last_became_peered":"2024-03-24T10:33:55.158875+0000","last_unstale":"2024-03-24T10:33:56.124458+0000","last_undegraded":"2024-03-24T10:33:56.124458+0000","last_fullsized":"2024-03-24T10:33:56.124458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123157+0000","last_change":"2024-03-24T10:33:55.134527+0000","last_active":"2024-03-24T10:33:56.123157+0000","last_peered":"2024-03-24T10:33:56.123157+0000","last_clean":"2024-03-24T10:33:56.123157+0000","last_became_active":"2024-03-24T10:33:55.134220+0000","last_became_peered":"2024-03-24T10:33:55.134220+0000","last_unstale":"2024-03-24T10:33:56.123157+0000","last_undegraded":"2024-03-24T10:33:56.123157+0000","last_fullsized":"2024-03-24T10:33:56.123157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306047+0000","last_change":"2024-03-24T10:33:55.138237+0000","last_active":"2024-03-24T10:33:56.306047+0000","last_peered":"2024-03-24T10:33:56.306047+0000","last_clean":"2024-03-24T10:33:56.306047+0000","last_became_active":"2024-03-24T10:33:55.136755+0000","last_became_peered":"2024-03-24T10:33:55.136755+0000","last_unstale":"2024-03-24T10:33:56.306047+0000","last_undegraded":"2024-03-24T10:33:56.306047+0000","last_fullsized":"2024-03-24T10:33:56.306047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305840+0000","last_change":"2024-03-24T10:33:55.145372+0000","last_active":"2024-03-24T10:33:56.305840+0000","last_peered":"2024-03-24T10:33:56.305840+0000","last_clean":"2024-03-24T10:33:56.305840+0000","last_became_active":"2024-03-24T10:33:55.144207+0000","last_became_peered":"2024-03-24T10:33:55.144207+0000","last_unstale":"2024-03-24T10:33:56.305840+0000","last_undegraded":"2024-03-24T10:33:56.305840+0000","last_fullsized":"2024-03-24T10:33:56.305840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:28.742456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306010+0000","last_change":"2024-03-24T10:33:55.146001+0000","last_active":"2024-03-24T10:33:56.306010+0000","last_peered":"2024-03-24T10:33:56.306010+0000","last_clean":"2024-03-24T10:33:56.306010+0000","last_became_active":"2024-03-24T10:33:55.145472+0000","last_became_peered":"2024-03-24T10:33:55.145472+0000","last_unstale":"2024-03-24T10:33:56.306010+0000","last_undegraded":"2024-03-24T10:33:56.306010+0000","last_fullsized":"2024-03-24T10:33:56.306010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:01.614093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123635+0000","last_change":"2024-03-24T10:33:55.143320+0000","last_active":"2024-03-24T10:33:56.123635+0000","last_peered":"2024-03-24T10:33:56.123635+0000","last_clean":"2024-03-24T10:33:56.123635+0000","last_became_active":"2024-03-24T10:33:55.143046+0000","last_became_peered":"2024-03-24T10:33:55.143046+0000","last_unstale":"2024-03-24T10:33:56.123635+0000","last_undegraded":"2024-03-24T10:33:56.123635+0000","last_fullsized":"2024-03-24T10:33:56.123635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124493+0000","last_change":"2024-03-24T10:33:55.147275+0000","last_active":"2024-03-24T10:33:56.124493+0000","last_peered":"2024-03-24T10:33:56.124493+0000","last_clean":"2024-03-24T10:33:56.124493+0000","last_became_active":"2024-03-24T10:33:55.146476+0000","last_became_peered":"2024-03-24T10:33:55.146476+0000","last_unstale":"2024-03-24T10:33:56.124493+0000","last_undegraded":"2024-03-24T10:33:56.124493+0000","last_fullsized":"2024-03-24T10:33:56.124493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305716+0000","last_change":"2024-03-24T10:33:55.131787+0000","last_active":"2024-03-24T10:33:56.305716+0000","last_peered":"2024-03-24T10:33:56.305716+0000","last_clean":"2024-03-24T10:33:56.305716+0000","last_became_active":"2024-03-24T10:33:55.131462+0000","last_became_peered":"2024-03-24T10:33:55.131462+0000","last_unstale":"2024-03-24T10:33:56.305716+0000","last_undegraded":"2024-03-24T10:33:56.305716+0000","last_fullsized":"2024-03-24T10:33:56.305716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:24.900927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305842+0000","last_change":"2024-03-24T10:33:54.130171+0000","last_active":"2024-03-24T10:33:56.305842+0000","last_peered":"2024-03-24T10:33:56.305842+0000","last_clean":"2024-03-24T10:33:56.305842+0000","last_became_active":"2024-03-24T10:33:54.129845+0000","last_became_peered":"2024-03-24T10:33:54.129845+0000","last_unstale":"2024-03-24T10:33:56.305842+0000","last_undegraded":"2024-03-24T10:33:56.305842+0000","last_fullsized":"2024-03-24T10:33:56.305842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305988+0000","last_change":"2024-03-24T10:33:55.156576+0000","last_active":"2024-03-24T10:33:56.305988+0000","last_peered":"2024-03-24T10:33:56.305988+0000","last_clean":"2024-03-24T10:33:56.305988+0000","last_became_active":"2024-03-24T10:33:55.156301+0000","last_became_peered":"2024-03-24T10:33:55.156301+0000","last_unstale":"2024-03-24T10:33:56.305988+0000","last_undegraded":"2024-03-24T10:33:56.305988+0000","last_fullsized":"2024-03-24T10:33:56.305988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124113+0000","last_change":"2024-03-24T10:33:54.122838+0000","last_active":"2024-03-24T10:33:56.124113+0000","last_peered":"2024-03-24T10:33:56.124113+0000","last_clean":"2024-03-24T10:33:56.124113+0000","last_became_active":"2024-03-24T10:33:54.122012+0000","last_became_peered":"2024-03-24T10:33:54.122012+0000","last_unstale":"2024-03-24T10:33:56.124113+0000","last_undegraded":"2024-03-24T10:33:56.124113+0000","last_fullsized":"2024-03-24T10:33:56.124113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305341+0000","last_change":"2024-03-24T10:33:54.127097+0000","last_active":"2024-03-24T10:33:56.305341+0000","last_peered":"2024-03-24T10:33:56.305341+0000","last_clean":"2024-03-24T10:33:56.305341+0000","last_became_active":"2024-03-24T10:33:54.126575+0000","last_became_peered":"2024-03-24T10:33:54.126575+0000","last_unstale":"2024-03-24T10:33:56.305341+0000","last_undegraded":"2024-03-24T10:33:56.305341+0000","last_fullsized":"2024-03-24T10:33:56.305341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305318+0000","last_change":"2024-03-24T10:33:55.138621+0000","last_active":"2024-03-24T10:33:56.305318+0000","last_peered":"2024-03-24T10:33:56.305318+0000","last_clean":"2024-03-24T10:33:56.305318+0000","last_became_active":"2024-03-24T10:33:55.138352+0000","last_became_peered":"2024-03-24T10:33:55.138352+0000","last_unstale":"2024-03-24T10:33:56.305318+0000","last_undegraded":"2024-03-24T10:33:56.305318+0000","last_fullsized":"2024-03-24T10:33:56.305318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:01.603353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305437+0000","last_change":"2024-03-24T10:33:54.125263+0000","last_active":"2024-03-24T10:33:56.305437+0000","last_peered":"2024-03-24T10:33:56.305437+0000","last_clean":"2024-03-24T10:33:56.305437+0000","last_became_active":"2024-03-24T10:33:54.124945+0000","last_became_peered":"2024-03-24T10:33:54.124945+0000","last_unstale":"2024-03-24T10:33:56.305437+0000","last_undegraded":"2024-03-24T10:33:56.305437+0000","last_fullsized":"2024-03-24T10:33:56.305437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122873+0000","last_change":"2024-03-24T10:33:55.134395+0000","last_active":"2024-03-24T10:33:56.122873+0000","last_peered":"2024-03-24T10:33:56.122873+0000","last_clean":"2024-03-24T10:33:56.122873+0000","last_became_active":"2024-03-24T10:33:55.133957+0000","last_became_peered":"2024-03-24T10:33:55.133957+0000","last_unstale":"2024-03-24T10:33:56.122873+0000","last_undegraded":"2024-03-24T10:33:56.122873+0000","last_fullsized":"2024-03-24T10:33:56.122873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124381+0000","last_change":"2024-03-24T10:33:54.127218+0000","last_active":"2024-03-24T10:33:56.124381+0000","last_peered":"2024-03-24T10:33:56.124381+0000","last_clean":"2024-03-24T10:33:56.124381+0000","last_became_active":"2024-03-24T10:33:54.126932+0000","last_became_peered":"2024-03-24T10:33:54.126932+0000","last_unstale":"2024-03-24T10:33:56.124381+0000","last_undegraded":"2024-03-24T10:33:56.124381+0000","last_fullsized":"2024-03-24T10:33:56.124381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306066+0000","last_change":"2024-03-24T10:33:55.150999+0000","last_active":"2024-03-24T10:33:56.306066+0000","last_peered":"2024-03-24T10:33:56.306066+0000","last_clean":"2024-03-24T10:33:56.306066+0000","last_became_active":"2024-03-24T10:33:55.150729+0000","last_became_peered":"2024-03-24T10:33:55.150729+0000","last_unstale":"2024-03-24T10:33:56.306066+0000","last_undegraded":"2024-03-24T10:33:56.306066+0000","last_fullsized":"2024-03-24T10:33:56.306066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306163+0000","last_change":"2024-03-24T10:33:54.126346+0000","last_active":"2024-03-24T10:33:56.306163+0000","last_peered":"2024-03-24T10:33:56.306163+0000","last_clean":"2024-03-24T10:33:56.306163+0000","last_became_active":"2024-03-24T10:33:54.126026+0000","last_became_peered":"2024-03-24T10:33:54.126026+0000","last_unstale":"2024-03-24T10:33:56.306163+0000","last_undegraded":"2024-03-24T10:33:56.306163+0000","last_fullsized":"2024-03-24T10:33:56.306163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306298+0000","last_change":"2024-03-24T10:33:55.145385+0000","last_active":"2024-03-24T10:33:56.306298+0000","last_peered":"2024-03-24T10:33:56.306298+0000","last_clean":"2024-03-24T10:33:56.306298+0000","last_became_active":"2024-03-24T10:33:55.144766+0000","last_became_peered":"2024-03-24T10:33:55.144766+0000","last_unstale":"2024-03-24T10:33:56.306298+0000","last_undegraded":"2024-03-24T10:33:56.306298+0000","last_fullsized":"2024-03-24T10:33:56.306298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123213+0000","last_change":"2024-03-24T10:33:54.128955+0000","last_active":"2024-03-24T10:33:56.123213+0000","last_peered":"2024-03-24T10:33:56.123213+0000","last_clean":"2024-03-24T10:33:56.123213+0000","last_became_active":"2024-03-24T10:33:54.128654+0000","last_became_peered":"2024-03-24T10:33:54.128654+0000","last_unstale":"2024-03-24T10:33:56.123213+0000","last_undegraded":"2024-03-24T10:33:56.123213+0000","last_fullsized":"2024-03-24T10:33:56.123213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306400+0000","last_change":"2024-03-24T10:33:55.146595+0000","last_active":"2024-03-24T10:33:56.306400+0000","last_peered":"2024-03-24T10:33:56.306400+0000","last_clean":"2024-03-24T10:33:56.306400+0000","last_became_active":"2024-03-24T10:33:55.146309+0000","last_became_peered":"2024-03-24T10:33:55.146309+0000","last_unstale":"2024-03-24T10:33:56.306400+0000","last_undegraded":"2024-03-24T10:33:56.306400+0000","last_fullsized":"2024-03-24T10:33:56.306400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.526613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124069+0000","last_change":"2024-03-24T10:33:54.125637+0000","last_active":"2024-03-24T10:33:56.124069+0000","last_peered":"2024-03-24T10:33:56.124069+0000","last_clean":"2024-03-24T10:33:56.124069+0000","last_became_active":"2024-03-24T10:33:54.125278+0000","last_became_peered":"2024-03-24T10:33:54.125278+0000","last_unstale":"2024-03-24T10:33:56.124069+0000","last_undegraded":"2024-03-24T10:33:56.124069+0000","last_fullsized":"2024-03-24T10:33:56.124069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306668+0000","last_change":"2024-03-24T10:33:55.156480+0000","last_active":"2024-03-24T10:33:56.306668+0000","last_peered":"2024-03-24T10:33:56.306668+0000","last_clean":"2024-03-24T10:33:56.306668+0000","last_became_active":"2024-03-24T10:33:55.156201+0000","last_became_peered":"2024-03-24T10:33:55.156201+0000","last_unstale":"2024-03-24T10:33:56.306668+0000","last_undegraded":"2024-03-24T10:33:56.306668+0000","last_fullsized":"2024-03-24T10:33:56.306668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:49.265965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306091+0000","last_change":"2024-03-24T10:33:54.125505+0000","last_active":"2024-03-24T10:33:56.306091+0000","last_peered":"2024-03-24T10:33:56.306091+0000","last_clean":"2024-03-24T10:33:56.306091+0000","last_became_active":"2024-03-24T10:33:54.125170+0000","last_became_peered":"2024-03-24T10:33:54.125170+0000","last_unstale":"2024-03-24T10:33:56.306091+0000","last_undegraded":"2024-03-24T10:33:56.306091+0000","last_fullsized":"2024-03-24T10:33:56.306091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305748+0000","last_change":"2024-03-24T10:33:55.156955+0000","last_active":"2024-03-24T10:33:56.305748+0000","last_peered":"2024-03-24T10:33:56.305748+0000","last_clean":"2024-03-24T10:33:56.305748+0000","last_became_active":"2024-03-24T10:33:55.156565+0000","last_became_peered":"2024-03-24T10:33:55.156565+0000","last_unstale":"2024-03-24T10:33:56.305748+0000","last_undegraded":"2024-03-24T10:33:56.305748+0000","last_fullsized":"2024-03-24T10:33:56.305748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:32.795374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306167+0000","last_change":"2024-03-24T10:33:54.128904+0000","last_active":"2024-03-24T10:33:56.306167+0000","last_peered":"2024-03-24T10:33:56.306167+0000","last_clean":"2024-03-24T10:33:56.306167+0000","last_became_active":"2024-03-24T10:33:54.128349+0000","last_became_peered":"2024-03-24T10:33:54.128349+0000","last_unstale":"2024-03-24T10:33:56.306167+0000","last_undegraded":"2024-03-24T10:33:56.306167+0000","last_fullsized":"2024-03-24T10:33:56.306167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306695+0000","last_change":"2024-03-24T10:33:55.155945+0000","last_active":"2024-03-24T10:33:56.306695+0000","last_peered":"2024-03-24T10:33:56.306695+0000","last_clean":"2024-03-24T10:33:56.306695+0000","last_became_active":"2024-03-24T10:33:55.155660+0000","last_became_peered":"2024-03-24T10:33:55.155660+0000","last_unstale":"2024-03-24T10:33:56.306695+0000","last_undegraded":"2024-03-24T10:33:56.306695+0000","last_fullsized":"2024-03-24T10:33:56.306695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124002+0000","last_change":"2024-03-24T10:33:54.127975+0000","last_active":"2024-03-24T10:33:56.124002+0000","last_peered":"2024-03-24T10:33:56.124002+0000","last_clean":"2024-03-24T10:33:56.124002+0000","last_became_active":"2024-03-24T10:33:54.127705+0000","last_became_peered":"2024-03-24T10:33:54.127705+0000","last_unstale":"2024-03-24T10:33:56.124002+0000","last_undegraded":"2024-03-24T10:33:56.124002+0000","last_fullsized":"2024-03-24T10:33:56.124002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124453+0000","last_change":"2024-03-24T10:33:55.146773+0000","last_active":"2024-03-24T10:33:56.124453+0000","last_peered":"2024-03-24T10:33:56.124453+0000","last_clean":"2024-03-24T10:33:56.124453+0000","last_became_active":"2024-03-24T10:33:55.146086+0000","last_became_peered":"2024-03-24T10:33:55.146086+0000","last_unstale":"2024-03-24T10:33:56.124453+0000","last_undegraded":"2024-03-24T10:33:56.124453+0000","last_fullsized":"2024-03-24T10:33:56.124453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123547+0000","last_change":"2024-03-24T10:33:54.114004+0000","last_active":"2024-03-24T10:33:56.123547+0000","last_peered":"2024-03-24T10:33:56.123547+0000","last_clean":"2024-03-24T10:33:56.123547+0000","last_became_active":"2024-03-24T10:33:54.113714+0000","last_became_peered":"2024-03-24T10:33:54.113714+0000","last_unstale":"2024-03-24T10:33:56.123547+0000","last_undegraded":"2024-03-24T10:33:56.123547+0000","last_fullsized":"2024-03-24T10:33:56.123547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:51.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306338+0000","last_change":"2024-03-24T10:33:55.160713+0000","last_active":"2024-03-24T10:33:56.306338+0000","last_peered":"2024-03-24T10:33:56.306338+0000","last_clean":"2024-03-24T10:33:56.306338+0000","last_became_active":"2024-03-24T10:33:55.157835+0000","last_became_peered":"2024-03-24T10:33:55.157835+0000","last_unstale":"2024-03-24T10:33:56.306338+0000","last_undegraded":"2024-03-24T10:33:56.306338+0000","last_fullsized":"2024-03-24T10:33:56.306338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:07.944846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306046+0000","last_change":"2024-03-24T10:33:54.123641+0000","last_active":"2024-03-24T10:33:56.306046+0000","last_peered":"2024-03-24T10:33:56.306046+0000","last_clean":"2024-03-24T10:33:56.306046+0000","last_became_active":"2024-03-24T10:33:54.123045+0000","last_became_peered":"2024-03-24T10:33:54.123045+0000","last_unstale":"2024-03-24T10:33:56.306046+0000","last_undegraded":"2024-03-24T10:33:56.306046+0000","last_fullsized":"2024-03-24T10:33:56.306046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:41:41.902140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306617+0000","last_change":"2024-03-24T10:33:55.155307+0000","last_active":"2024-03-24T10:33:56.306617+0000","last_peered":"2024-03-24T10:33:56.306617+0000","last_clean":"2024-03-24T10:33:56.306617+0000","last_became_active":"2024-03-24T10:33:55.154976+0000","last_became_peered":"2024-03-24T10:33:55.154976+0000","last_unstale":"2024-03-24T10:33:56.306617+0000","last_undegraded":"2024-03-24T10:33:56.306617+0000","last_fullsized":"2024-03-24T10:33:56.306617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123767+0000","last_change":"2024-03-24T10:33:54.124359+0000","last_active":"2024-03-24T10:33:56.123767+0000","last_peered":"2024-03-24T10:33:56.123767+0000","last_clean":"2024-03-24T10:33:56.123767+0000","last_became_active":"2024-03-24T10:33:54.124063+0000","last_became_peered":"2024-03-24T10:33:54.124063+0000","last_unstale":"2024-03-24T10:33:56.123767+0000","last_undegraded":"2024-03-24T10:33:56.123767+0000","last_fullsized":"2024-03-24T10:33:56.123767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:15.778769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306899+0000","last_change":"2024-03-24T10:33:55.159546+0000","last_active":"2024-03-24T10:33:56.306899+0000","last_peered":"2024-03-24T10:33:56.306899+0000","last_clean":"2024-03-24T10:33:56.306899+0000","last_became_active":"2024-03-24T10:33:55.159193+0000","last_became_peered":"2024-03-24T10:33:55.159193+0000","last_unstale":"2024-03-24T10:33:56.306899+0000","last_undegraded":"2024-03-24T10:33:56.306899+0000","last_fullsized":"2024-03-24T10:33:56.306899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:07.558259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123249+0000","last_change":"2024-03-24T10:33:54.127205+0000","last_active":"2024-03-24T10:33:56.123249+0000","last_peered":"2024-03-24T10:33:56.123249+0000","last_clean":"2024-03-24T10:33:56.123249+0000","last_became_active":"2024-03-24T10:33:54.126938+0000","last_became_peered":"2024-03-24T10:33:54.126938+0000","last_unstale":"2024-03-24T10:33:56.123249+0000","last_undegraded":"2024-03-24T10:33:56.123249+0000","last_fullsized":"2024-03-24T10:33:56.123249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305970+0000","last_change":"2024-03-24T10:33:55.150130+0000","last_active":"2024-03-24T10:33:56.305970+0000","last_peered":"2024-03-24T10:33:56.305970+0000","last_clean":"2024-03-24T10:33:56.305970+0000","last_became_active":"2024-03-24T10:33:55.145689+0000","last_became_peered":"2024-03-24T10:33:55.145689+0000","last_unstale":"2024-03-24T10:33:56.305970+0000","last_undegraded":"2024-03-24T10:33:56.305970+0000","last_fullsized":"2024-03-24T10:33:56.305970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:56.746149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123759+0000","last_change":"2024-03-24T10:33:54.131439+0000","last_active":"2024-03-24T10:33:56.123759+0000","last_peered":"2024-03-24T10:33:56.123759+0000","last_clean":"2024-03-24T10:33:56.123759+0000","last_became_active":"2024-03-24T10:33:54.131117+0000","last_became_peered":"2024-03-24T10:33:54.131117+0000","last_unstale":"2024-03-24T10:33:56.123759+0000","last_undegraded":"2024-03-24T10:33:56.123759+0000","last_fullsized":"2024-03-24T10:33:56.123759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306791+0000","last_change":"2024-03-24T10:33:55.157088+0000","last_active":"2024-03-24T10:33:56.306791+0000","last_peered":"2024-03-24T10:33:56.306791+0000","last_clean":"2024-03-24T10:33:56.306791+0000","last_became_active":"2024-03-24T10:33:55.156790+0000","last_became_peered":"2024-03-24T10:33:55.156790+0000","last_unstale":"2024-03-24T10:33:56.306791+0000","last_undegraded":"2024-03-24T10:33:56.306791+0000","last_fullsized":"2024-03-24T10:33:56.306791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:10:08.838145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306529+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:33:56.306529+0000","last_peered":"2024-03-24T10:33:56.306529+0000","last_clean":"2024-03-24T10:33:56.306529+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:33:56.306529+0000","last_undegraded":"2024-03-24T10:33:56.306529+0000","last_fullsized":"2024-03-24T10:33:56.306529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305276+0000","last_change":"2024-03-24T10:33:54.121820+0000","last_active":"2024-03-24T10:33:56.305276+0000","last_peered":"2024-03-24T10:33:56.305276+0000","last_clean":"2024-03-24T10:33:56.305276+0000","last_became_active":"2024-03-24T10:33:54.121496+0000","last_became_peered":"2024-03-24T10:33:54.121496+0000","last_unstale":"2024-03-24T10:33:56.305276+0000","last_undegraded":"2024-03-24T10:33:56.305276+0000","last_fullsized":"2024-03-24T10:33:56.305276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305400+0000","last_change":"2024-03-24T10:33:55.146810+0000","last_active":"2024-03-24T10:33:56.305400+0000","last_peered":"2024-03-24T10:33:56.305400+0000","last_clean":"2024-03-24T10:33:56.305400+0000","last_became_active":"2024-03-24T10:33:55.146508+0000","last_became_peered":"2024-03-24T10:33:55.146508+0000","last_unstale":"2024-03-24T10:33:56.305400+0000","last_undegraded":"2024-03-24T10:33:56.305400+0000","last_fullsized":"2024-03-24T10:33:56.305400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306540+0000","last_change":"2024-03-24T10:33:54.127727+0000","last_active":"2024-03-24T10:33:56.306540+0000","last_peered":"2024-03-24T10:33:56.306540+0000","last_clean":"2024-03-24T10:33:56.306540+0000","last_became_active":"2024-03-24T10:33:54.127106+0000","last_became_peered":"2024-03-24T10:33:54.127106+0000","last_unstale":"2024-03-24T10:33:56.306540+0000","last_undegraded":"2024-03-24T10:33:56.306540+0000","last_fullsized":"2024-03-24T10:33:56.306540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:31.791505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306196+0000","last_change":"2024-03-24T10:33:55.156425+0000","last_active":"2024-03-24T10:33:56.306196+0000","last_peered":"2024-03-24T10:33:56.306196+0000","last_clean":"2024-03-24T10:33:56.306196+0000","last_became_active":"2024-03-24T10:33:55.156146+0000","last_became_peered":"2024-03-24T10:33:55.156146+0000","last_unstale":"2024-03-24T10:33:56.306196+0000","last_undegraded":"2024-03-24T10:33:56.306196+0000","last_fullsized":"2024-03-24T10:33:56.306196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123643+0000","last_change":"2024-03-24T10:33:54.127887+0000","last_active":"2024-03-24T10:33:56.123643+0000","last_peered":"2024-03-24T10:33:56.123643+0000","last_clean":"2024-03-24T10:33:56.123643+0000","last_became_active":"2024-03-24T10:33:54.127602+0000","last_became_peered":"2024-03-24T10:33:54.127602+0000","last_unstale":"2024-03-24T10:33:56.123643+0000","last_undegraded":"2024-03-24T10:33:56.123643+0000","last_fullsized":"2024-03-24T10:33:56.123643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122824+0000","last_change":"2024-03-24T10:33:55.128811+0000","last_active":"2024-03-24T10:33:56.122824+0000","last_peered":"2024-03-24T10:33:56.122824+0000","last_clean":"2024-03-24T10:33:56.122824+0000","last_became_active":"2024-03-24T10:33:55.128094+0000","last_became_peered":"2024-03-24T10:33:55.128094+0000","last_unstale":"2024-03-24T10:33:56.122824+0000","last_undegraded":"2024-03-24T10:33:56.122824+0000","last_fullsized":"2024-03-24T10:33:56.122824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306104+0000","last_change":"2024-03-24T10:33:54.123812+0000","last_active":"2024-03-24T10:33:56.306104+0000","last_peered":"2024-03-24T10:33:56.306104+0000","last_clean":"2024-03-24T10:33:56.306104+0000","last_became_active":"2024-03-24T10:33:54.122614+0000","last_became_peered":"2024-03-24T10:33:54.122614+0000","last_unstale":"2024-03-24T10:33:56.306104+0000","last_undegraded":"2024-03-24T10:33:56.306104+0000","last_fullsized":"2024-03-24T10:33:56.306104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305946+0000","last_change":"2024-03-24T10:33:55.140461+0000","last_active":"2024-03-24T10:33:56.305946+0000","last_peered":"2024-03-24T10:33:56.305946+0000","last_clean":"2024-03-24T10:33:56.305946+0000","last_became_active":"2024-03-24T10:33:55.139737+0000","last_became_peered":"2024-03-24T10:33:55.139737+0000","last_unstale":"2024-03-24T10:33:56.305946+0000","last_undegraded":"2024-03-24T10:33:56.305946+0000","last_fullsized":"2024-03-24T10:33:56.305946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:27.228493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122944+0000","last_change":"2024-03-24T10:33:54.123506+0000","last_active":"2024-03-24T10:33:56.122944+0000","last_peered":"2024-03-24T10:33:56.122944+0000","last_clean":"2024-03-24T10:33:56.122944+0000","last_became_active":"2024-03-24T10:33:54.123194+0000","last_became_peered":"2024-03-24T10:33:54.123194+0000","last_unstale":"2024-03-24T10:33:56.122944+0000","last_undegraded":"2024-03-24T10:33:56.122944+0000","last_fullsized":"2024-03-24T10:33:56.122944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:23.897057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305900+0000","last_change":"2024-03-24T10:33:55.148696+0000","last_active":"2024-03-24T10:33:56.305900+0000","last_peered":"2024-03-24T10:33:56.305900+0000","last_clean":"2024-03-24T10:33:56.305900+0000","last_became_active":"2024-03-24T10:33:55.148382+0000","last_became_peered":"2024-03-24T10:33:55.148382+0000","last_unstale":"2024-03-24T10:33:56.305900+0000","last_undegraded":"2024-03-24T10:33:56.305900+0000","last_fullsized":"2024-03-24T10:33:56.305900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:44:35.973234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306970+0000","last_change":"2024-03-24T10:33:54.129296+0000","last_active":"2024-03-24T10:33:56.306970+0000","last_peered":"2024-03-24T10:33:56.306970+0000","last_clean":"2024-03-24T10:33:56.306970+0000","last_became_active":"2024-03-24T10:33:54.128788+0000","last_became_peered":"2024-03-24T10:33:54.128788+0000","last_unstale":"2024-03-24T10:33:56.306970+0000","last_undegraded":"2024-03-24T10:33:56.306970+0000","last_fullsized":"2024-03-24T10:33:56.306970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:48.412131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122832+0000","last_change":"2024-03-24T10:33:55.128461+0000","last_active":"2024-03-24T10:33:56.122832+0000","last_peered":"2024-03-24T10:33:56.122832+0000","last_clean":"2024-03-24T10:33:56.122832+0000","last_became_active":"2024-03-24T10:33:55.128125+0000","last_became_peered":"2024-03-24T10:33:55.128125+0000","last_unstale":"2024-03-24T10:33:56.122832+0000","last_undegraded":"2024-03-24T10:33:56.122832+0000","last_fullsized":"2024-03-24T10:33:56.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:26.312372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.140759+0000","last_change":"2024-03-24T10:33:54.117337+0000","last_active":"2024-03-24T10:33:56.140759+0000","last_peered":"2024-03-24T10:33:56.140759+0000","last_clean":"2024-03-24T10:33:56.140759+0000","last_became_active":"2024-03-24T10:33:54.116814+0000","last_became_peered":"2024-03-24T10:33:54.116814+0000","last_unstale":"2024-03-24T10:33:56.140759+0000","last_undegraded":"2024-03-24T10:33:56.140759+0000","last_fullsized":"2024-03-24T10:33:56.140759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305977+0000","last_change":"2024-03-24T10:33:55.147551+0000","last_active":"2024-03-24T10:33:56.305977+0000","last_peered":"2024-03-24T10:33:56.305977+0000","last_clean":"2024-03-24T10:33:56.305977+0000","last_became_active":"2024-03-24T10:33:55.147261+0000","last_became_peered":"2024-03-24T10:33:55.147261+0000","last_unstale":"2024-03-24T10:33:56.305977+0000","last_undegraded":"2024-03-24T10:33:56.305977+0000","last_fullsized":"2024-03-24T10:33:56.305977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:30.086948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.124432+0000","last_change":"2024-03-24T10:33:54.127434+0000","last_active":"2024-03-24T10:33:56.124432+0000","last_peered":"2024-03-24T10:33:56.124432+0000","last_clean":"2024-03-24T10:33:56.124432+0000","last_became_active":"2024-03-24T10:33:54.127162+0000","last_became_peered":"2024-03-24T10:33:54.127162+0000","last_unstale":"2024-03-24T10:33:56.124432+0000","last_undegraded":"2024-03-24T10:33:56.124432+0000","last_fullsized":"2024-03-24T10:33:56.124432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:21.592251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306039+0000","last_change":"2024-03-24T10:33:55.142227+0000","last_active":"2024-03-24T10:33:56.306039+0000","last_peered":"2024-03-24T10:33:56.306039+0000","last_clean":"2024-03-24T10:33:56.306039+0000","last_became_active":"2024-03-24T10:33:55.141920+0000","last_became_peered":"2024-03-24T10:33:55.141920+0000","last_unstale":"2024-03-24T10:33:56.306039+0000","last_undegraded":"2024-03-24T10:33:56.306039+0000","last_fullsized":"2024-03-24T10:33:56.306039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:14.871727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306452+0000","last_change":"2024-03-24T10:33:54.128286+0000","last_active":"2024-03-24T10:33:56.306452+0000","last_peered":"2024-03-24T10:33:56.306452+0000","last_clean":"2024-03-24T10:33:56.306452+0000","last_became_active":"2024-03-24T10:33:54.127434+0000","last_became_peered":"2024-03-24T10:33:54.127434+0000","last_unstale":"2024-03-24T10:33:56.306452+0000","last_undegraded":"2024-03-24T10:33:56.306452+0000","last_fullsized":"2024-03-24T10:33:56.306452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:40.602694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305662+0000","last_change":"2024-03-24T10:33:55.132582+0000","last_active":"2024-03-24T10:33:56.305662+0000","last_peered":"2024-03-24T10:33:56.305662+0000","last_clean":"2024-03-24T10:33:56.305662+0000","last_became_active":"2024-03-24T10:33:55.132259+0000","last_became_peered":"2024-03-24T10:33:55.132259+0000","last_unstale":"2024-03-24T10:33:56.305662+0000","last_undegraded":"2024-03-24T10:33:56.305662+0000","last_fullsized":"2024-03-24T10:33:56.305662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:53:22.596121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.122894+0000","last_change":"2024-03-24T10:33:54.127899+0000","last_active":"2024-03-24T10:33:56.122894+0000","last_peered":"2024-03-24T10:33:56.122894+0000","last_clean":"2024-03-24T10:33:56.122894+0000","last_became_active":"2024-03-24T10:33:54.127592+0000","last_became_peered":"2024-03-24T10:33:54.127592+0000","last_unstale":"2024-03-24T10:33:56.122894+0000","last_undegraded":"2024-03-24T10:33:56.122894+0000","last_fullsized":"2024-03-24T10:33:56.122894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:19.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305592+0000","last_change":"2024-03-24T10:33:55.133853+0000","last_active":"2024-03-24T10:33:56.305592+0000","last_peered":"2024-03-24T10:33:56.305592+0000","last_clean":"2024-03-24T10:33:56.305592+0000","last_became_active":"2024-03-24T10:33:55.133478+0000","last_became_peered":"2024-03-24T10:33:55.133478+0000","last_unstale":"2024-03-24T10:33:56.305592+0000","last_undegraded":"2024-03-24T10:33:56.305592+0000","last_fullsized":"2024-03-24T10:33:56.305592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123001+0000","last_change":"2024-03-24T10:33:54.126665+0000","last_active":"2024-03-24T10:33:56.123001+0000","last_peered":"2024-03-24T10:33:56.123001+0000","last_clean":"2024-03-24T10:33:56.123001+0000","last_became_active":"2024-03-24T10:33:54.126339+0000","last_became_peered":"2024-03-24T10:33:54.126339+0000","last_unstale":"2024-03-24T10:33:56.123001+0000","last_undegraded":"2024-03-24T10:33:56.123001+0000","last_fullsized":"2024-03-24T10:33:56.123001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:26.224623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305669+0000","last_change":"2024-03-24T10:33:55.140585+0000","last_active":"2024-03-24T10:33:56.305669+0000","last_peered":"2024-03-24T10:33:56.305669+0000","last_clean":"2024-03-24T10:33:56.305669+0000","last_became_active":"2024-03-24T10:33:55.140142+0000","last_became_peered":"2024-03-24T10:33:55.140142+0000","last_unstale":"2024-03-24T10:33:56.305669+0000","last_undegraded":"2024-03-24T10:33:56.305669+0000","last_fullsized":"2024-03-24T10:33:56.305669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:06.836936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306757+0000","last_change":"2024-03-24T10:33:54.128102+0000","last_active":"2024-03-24T10:33:56.306757+0000","last_peered":"2024-03-24T10:33:56.306757+0000","last_clean":"2024-03-24T10:33:56.306757+0000","last_became_active":"2024-03-24T10:33:54.127077+0000","last_became_peered":"2024-03-24T10:33:54.127077+0000","last_unstale":"2024-03-24T10:33:56.306757+0000","last_undegraded":"2024-03-24T10:33:56.306757+0000","last_fullsized":"2024-03-24T10:33:56.306757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:20:32.560323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.123921+0000","last_change":"2024-03-24T10:33:55.155558+0000","last_active":"2024-03-24T10:33:56.123921+0000","last_peered":"2024-03-24T10:33:56.123921+0000","last_clean":"2024-03-24T10:33:56.123921+0000","last_became_active":"2024-03-24T10:33:55.155291+0000","last_became_peered":"2024-03-24T10:33:55.155291+0000","last_unstale":"2024-03-24T10:33:56.123921+0000","last_undegraded":"2024-03-24T10:33:56.123921+0000","last_fullsized":"2024-03-24T10:33:56.123921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:17.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305743+0000","last_change":"2024-03-24T10:33:55.159176+0000","last_active":"2024-03-24T10:33:56.305743+0000","last_peered":"2024-03-24T10:33:56.305743+0000","last_clean":"2024-03-24T10:33:56.305743+0000","last_became_active":"2024-03-24T10:33:55.158885+0000","last_became_peered":"2024-03-24T10:33:55.158885+0000","last_unstale":"2024-03-24T10:33:56.305743+0000","last_undegraded":"2024-03-24T10:33:56.305743+0000","last_fullsized":"2024-03-24T10:33:56.305743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:36.805469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306353+0000","last_change":"2024-03-24T10:33:54.132298+0000","last_active":"2024-03-24T10:33:56.306353+0000","last_peered":"2024-03-24T10:33:56.306353+0000","last_clean":"2024-03-24T10:33:56.306353+0000","last_became_active":"2024-03-24T10:33:54.132009+0000","last_became_peered":"2024-03-24T10:33:54.132009+0000","last_unstale":"2024-03-24T10:33:56.306353+0000","last_undegraded":"2024-03-24T10:33:56.306353+0000","last_fullsized":"2024-03-24T10:33:56.306353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:04.522743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305825+0000","last_change":"2024-03-24T10:33:55.159162+0000","last_active":"2024-03-24T10:33:56.305825+0000","last_peered":"2024-03-24T10:33:56.305825+0000","last_clean":"2024-03-24T10:33:56.305825+0000","last_became_active":"2024-03-24T10:33:55.158898+0000","last_became_peered":"2024-03-24T10:33:55.158898+0000","last_unstale":"2024-03-24T10:33:56.305825+0000","last_undegraded":"2024-03-24T10:33:56.305825+0000","last_fullsized":"2024-03-24T10:33:56.305825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:07.263093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306863+0000","last_change":"2024-03-24T10:33:54.129068+0000","last_active":"2024-03-24T10:33:56.306863+0000","last_peered":"2024-03-24T10:33:56.306863+0000","last_clean":"2024-03-24T10:33:56.306863+0000","last_became_active":"2024-03-24T10:33:54.128465+0000","last_became_peered":"2024-03-24T10:33:54.128465+0000","last_unstale":"2024-03-24T10:33:56.306863+0000","last_undegraded":"2024-03-24T10:33:56.306863+0000","last_fullsized":"2024-03-24T10:33:56.306863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:25.221403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306248+0000","last_change":"2024-03-24T10:33:55.146210+0000","last_active":"2024-03-24T10:33:56.306248+0000","last_peered":"2024-03-24T10:33:56.306248+0000","last_clean":"2024-03-24T10:33:56.306248+0000","last_became_active":"2024-03-24T10:33:55.145069+0000","last_became_peered":"2024-03-24T10:33:55.145069+0000","last_unstale":"2024-03-24T10:33:56.306248+0000","last_undegraded":"2024-03-24T10:33:56.306248+0000","last_fullsized":"2024-03-24T10:33:56.306248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:05.366042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305927+0000","last_change":"2024-03-24T10:33:54.126679+0000","last_active":"2024-03-24T10:33:56.305927+0000","last_peered":"2024-03-24T10:33:56.305927+0000","last_clean":"2024-03-24T10:33:56.305927+0000","last_became_active":"2024-03-24T10:33:54.126407+0000","last_became_peered":"2024-03-24T10:33:54.126407+0000","last_unstale":"2024-03-24T10:33:56.305927+0000","last_undegraded":"2024-03-24T10:33:56.305927+0000","last_fullsized":"2024-03-24T10:33:56.305927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:53:40.115020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.305615+0000","last_change":"2024-03-24T10:33:55.150245+0000","last_active":"2024-03-24T10:33:56.305615+0000","last_peered":"2024-03-24T10:33:56.305615+0000","last_clean":"2024-03-24T10:33:56.305615+0000","last_became_active":"2024-03-24T10:33:55.149927+0000","last_became_peered":"2024-03-24T10:33:55.149927+0000","last_unstale":"2024-03-24T10:33:56.305615+0000","last_undegraded":"2024-03-24T10:33:56.305615+0000","last_fullsized":"2024-03-24T10:33:56.305615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:54.104706+0000","last_clean_scrub_stamp":"2024-03-24T10:33:54.104706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:55.789183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:33:56.306337+0000","last_change":"2024-03-24T10:33:54.127624+0000","last_active":"2024-03-24T10:33:56.306337+0000","last_peered":"2024-03-24T10:33:56.306337+0000","last_clean":"2024-03-24T10:33:56.306337+0000","last_became_active":"2024-03-24T10:33:54.127307+0000","last_became_peered":"2024-03-24T10:33:54.127307+0000","last_unstale":"2024-03-24T10:33:56.306337+0000","last_undegraded":"2024-03-24T10:33:56.306337+0000","last_fullsized":"2024-03-24T10:33:56.306337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:53.100837+0000","last_clean_scrub_stamp":"2024-03-24T10:33:53.100837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:12.673017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:34:05.379 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T10:34:05.379 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T10:34:05.380 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-24T10:34:05.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T10:34:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.513+0000 7f09b9c15640 1 Processor -- start 2024-03-24T10:34:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.513+0000 7f09b9c15640 1 -- start start 2024-03-24T10:34:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b9c15640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 0x7f09b40afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b9c15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09b40affe0 0x7f09b406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b9c15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b415f5e0 con 0x7f09b406b730 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b9c15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f09b406dbf0 con 0x7f09b40affe0 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b9c15640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f09b40b0470 con 0x7f09b415bb30 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09b40affe0 0x7f09b406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f09b406b730 0x7f09b4071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41680/0 (socket says 172.21.15.1:41680) 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09b40affe0 0x7f09b406b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33872/0 (socket says 172.21.15.1:33872) 2024-03-24T10:34:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b3fff640 1 -- 172.21.15.1:0/2886721560 learned_addr learned my addr 172.21.15.1:0/2886721560 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b37fe640 1 --2- 172.21.15.1:0/2886721560 >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 0x7f09b40afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 -- 172.21.15.1:0/2886721560 >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 msgr2=0x7f09b40afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 --2- 172.21.15.1:0/2886721560 >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 0x7f09b40afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 -- 172.21.15.1:0/2886721560 >> v1:172.21.15.1:6789/0 conn(0x7f09b406b730 legacy=0x7f09b4071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b2ffd640 1 -- 172.21.15.1:0/2886721560 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b406bf60 con 0x7f09b40affe0 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.514+0000 7f09b27fc640 1 -- 172.21.15.1:0/2886721560 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722508377 0 0) 0x7f09b415f5e0 con 0x7f09b406b730 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.515+0000 7f09b37fe640 1 --2- 172.21.15.1:0/2886721560 >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 0x7f09b40afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.515+0000 7f09b2ffd640 1 --2- 172.21.15.1:0/2886721560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b40affe0 0x7f09b406b1f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09b406d780 tx=0x7f09a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17428cf7c7961d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.515+0000 7f09b27fc640 1 -- 172.21.15.1:0/2886721560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4031830 con 0x7f09b40affe0 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.515+0000 7f09b27fc640 1 -- 172.21.15.1:0/2886721560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a4031990 con 0x7f09b40affe0 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.515+0000 7f09b27fc640 1 -- 172.21.15.1:0/2886721560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4031d20 con 0x7f09b40affe0 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 -- 172.21.15.1:0/2886721560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b40affe0 msgr2=0x7f09b406b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2886721560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b40affe0 0x7f09b406b1f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09b406d780 tx=0x7f09a402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 -- 172.21.15.1:0/2886721560 shutdown_connections 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2886721560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b40affe0 0x7f09b406b1f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2886721560 >> v2:172.21.15.3:3300/0 conn(0x7f09b415bb30 0x7f09b40afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 -- 172.21.15.1:0/2886721560 >> 172.21.15.1:0/2886721560 conn(0x7f09b40b27b0 msgr2=0x7f09b40b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 -- 172.21.15.1:0/2886721560 shutdown_connections 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.516+0000 7f09b9c15640 1 -- 172.21.15.1:0/2886721560 wait complete. 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.517+0000 7f09b9c15640 1 Processor -- start 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.517+0000 7f09b9c15640 1 -- start start 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09b406b730 0x7f09b415d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b40affe0 0x7f09b4160ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4161000 con 0x7f09b415bb30 2024-03-24T10:34:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f09b415d8a0 con 0x7f09b40affe0 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b9c15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f09b415db80 con 0x7f09b406b730 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33876/0 (socket says 172.21.15.1:33876) 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 -- 172.21.15.1:0/2452236683 learned_addr learned my addr 172.21.15.1:0/2452236683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b37fe640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09b406b730 0x7f09b415d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b2ffd640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b40affe0 0x7f09b4160ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 -- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09b406b730 msgr2=0x7f09b415d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09b406b730 0x7f09b415d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 -- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b40affe0 msgr2=0x7f09b4160ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b40affe0 0x7f09b4160ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.518+0000 7f09b3fff640 1 -- 172.21.15.1:0/2452236683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a8009400 con 0x7f09b415bb30 2024-03-24T10:34:05.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f09b3fff640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c7a0 tx=0x7f09ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac00a640 con 0x7f09b415bb30 2024-03-24T10:34:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ac00a7a0 con 0x7f09b415bb30 2024-03-24T10:34:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a4009400 con 0x7f09b415bb30 2024-03-24T10:34:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac00abd0 con 0x7f09b415bb30 2024-03-24T10:34:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.519+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b41541c0 con 0x7f09b415bb30 2024-03-24T10:34:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.520+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b4075470 con 0x7f09b415bb30 2024-03-24T10:34:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.520+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09ac00ad30 con 0x7f09b415bb30 2024-03-24T10:34:05.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.521+0000 7f099bfff640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 0x7f09a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.521+0000 7f09b37fe640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 0x7f09a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.521+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f09ac0899a0 con 0x7f09b415bb30 2024-03-24T10:34:05.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.521+0000 7f09b37fe640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 0x7f09a006dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009c30 tx=0x7f09a8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.525+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09ac054470 con 0x7f09b415bb30 2024-03-24T10:34:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.743+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f09b4153d40 con 0x7f09b415bb30 2024-03-24T10:34:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.743+0000 7f099bfff640 1 -- 172.21.15.1:0/2452236683 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f09b4153d40 con 0x7f09b415bb30 2024-03-24T10:34:05.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:05.745 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 msgr2=0x7f09a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 0x7f09a006dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009c30 tx=0x7f09a8009190 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 msgr2=0x7f09b414c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c7a0 tx=0x7f09ac00cc60 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 shutdown_connections 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f09a006b810 0x7f09a006dcd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09b415bb30 0x7f09b414c640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b40affe0 0x7f09b4160ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 --2- 172.21.15.1:0/2452236683 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09b406b730 0x7f09b415d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 >> 172.21.15.1:0/2452236683 conn(0x7f09b40b27b0 msgr2=0x7f09b406eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 shutdown_connections 2024-03-24T10:34:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.745+0000 7f09b9c15640 1 -- 172.21.15.1:0/2452236683 wait complete. 2024-03-24T10:34:05.757 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-24T10:34:05.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.858+0000 7fb61958a640 1 Processor -- start 2024-03-24T10:34:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.858+0000 7fb61958a640 1 -- start start 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb61958a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 0x7fb6140d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb61958a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 0x7fb6140d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb61958a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6140b19b0 con 0x7fb6140b1e40 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb61958a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb6140d0070 con 0x7fb61415c060 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb61958a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb6140cfdb0 con 0x7fb6140d27e0 2024-03-24T10:34:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb6137fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb6140b1e40 0x7fb6140b22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41694/0 (socket says 172.21.15.1:41694) 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb6137fe640 1 -- 172.21.15.1:0/4034501276 learned_addr learned my addr 172.21.15.1:0/4034501276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb6127fc640 1 --2- 172.21.15.1:0/4034501276 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 0x7fb6140d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.859+0000 7fb612ffd640 1 --2- 172.21.15.1:0/4034501276 >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 0x7fb6140d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb611ffb640 1 -- 172.21.15.1:0/4034501276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347579052 0 0) 0x7fb6140b19b0 con 0x7fb6140b1e40 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb6127fc640 1 -- 172.21.15.1:0/4034501276 >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 msgr2=0x7fb6140d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb6127fc640 1 --2- 172.21.15.1:0/4034501276 >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 0x7fb6140d02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb6127fc640 1 -- 172.21.15.1:0/4034501276 >> v1:172.21.15.1:6789/0 conn(0x7fb6140b1e40 legacy=0x7fb6140b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb6127fc640 1 -- 172.21.15.1:0/4034501276 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6140b1360 con 0x7fb6140d27e0 2024-03-24T10:34:05.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb612ffd640 1 --2- 172.21.15.1:0/4034501276 >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 0x7fb6140d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.860+0000 7fb6127fc640 1 --2- 172.21.15.1:0/4034501276 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 0x7fb6140d07e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0097a0 tx=0x7fb5fc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d2a1b66da4d1eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb611ffb640 1 -- 172.21.15.1:0/4034501276 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc02f050 con 0x7fb6140d27e0 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb611ffb640 1 -- 172.21.15.1:0/4034501276 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5fc031700 con 0x7fb6140d27e0 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb611ffb640 1 -- 172.21.15.1:0/4034501276 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc031a90 con 0x7fb6140d27e0 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 -- 172.21.15.1:0/4034501276 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 msgr2=0x7fb6140d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 --2- 172.21.15.1:0/4034501276 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 0x7fb6140d07e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0097a0 tx=0x7fb5fc02b7a0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 -- 172.21.15.1:0/4034501276 shutdown_connections 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 --2- 172.21.15.1:0/4034501276 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb6140d27e0 0x7fb6140d07e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 --2- 172.21.15.1:0/4034501276 >> v2:172.21.15.3:3300/0 conn(0x7fb61415c060 0x7fb6140d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 -- 172.21.15.1:0/4034501276 >> 172.21.15.1:0/4034501276 conn(0x7fb6140d1cc0 msgr2=0x7fb6140d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 -- 172.21.15.1:0/4034501276 shutdown_connections 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.861+0000 7fb61958a640 1 -- 172.21.15.1:0/4034501276 wait complete. 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.862+0000 7fb61958a640 1 Processor -- start 2024-03-24T10:34:05.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 -- start start 2024-03-24T10:34:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6140d27e0 0x7fb6140b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb61415c060 0x7fb6140b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb614149110 con 0x7fb6140b1e40 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb6140b63b0 con 0x7fb6140d27e0 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb61958a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb6140b6690 con 0x7fb61415c060 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb612ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb612ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33880/0 (socket says 172.21.15.1:33880) 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb612ffd640 1 -- 172.21.15.1:0/2751864389 learned_addr learned my addr 172.21.15.1:0/2751864389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb6137fe640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb61415c060 0x7fb6140b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.863+0000 7fb6127fc640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6140d27e0 0x7fb6140b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 -- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb61415c060 msgr2=0x7fb6140b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb61415c060 0x7fb6140b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 -- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6140d27e0 msgr2=0x7fb6140b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6140d27e0 0x7fb6140b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 -- 172.21.15.1:0/2751864389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c009400 con 0x7fb6140b1e40 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb612ffd640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00ebe0 tx=0x7fb60c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c009760 con 0x7fb6140b1e40 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5fc009400 con 0x7fb6140b1e40 2024-03-24T10:34:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb60c00cc30 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c00eee0 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.864+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61415e060 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.865+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61404f4e0 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.865+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb60c013470 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.866+0000 7fb60b7fe640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 0x7fb5f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.866+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb60c08a4c0 con 0x7fb6140b1e40 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.866+0000 7fb6127fc640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 0x7fb5f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.866+0000 7fb6127fc640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 0x7fb5f406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc035000 tx=0x7fb5fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:05.870+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb60c054f10 con 0x7fb6140b1e40 2024-03-24T10:34:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.052+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6140b1010 con 0x7fb6140b1e40 2024-03-24T10:34:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.054+0000 7fb60b7fe640 1 -- 172.21.15.1:0/2751864389 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fb60c0585d0 con 0x7fb6140b1e40 2024-03-24T10:34:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.055+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 msgr2=0x7fb5f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 0x7fb5f406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc035000 tx=0x7fb5fc008030 comp rx=0 tx=0).stop 2024-03-24T10:34:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 msgr2=0x7fb614148a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00ebe0 tx=0x7fb60c00c520 comp rx=0 tx=0).stop 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 shutdown_connections 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb5f406b810 0x7fb5f406dcd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb61415c060 0x7fb6140b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6140d27e0 0x7fb6140b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 --2- 172.21.15.1:0/2751864389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6140b1e40 0x7fb614148a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 >> 172.21.15.1:0/2751864389 conn(0x7fb6140d1cc0 msgr2=0x7fb6140cd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 shutdown_connections 2024-03-24T10:34:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.056+0000 7fb61958a640 1 -- 172.21.15.1:0/2751864389 wait complete. 2024-03-24T10:34:06.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:06.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:06.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.171+0000 7f1887240640 1 Processor -- start 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.171+0000 7f1887240640 1 -- start start 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1887240640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 0x7f18800b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1887240640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18800ceb00 0x7f18800d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1887240640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18800cfef0 con 0x7f18800d0d00 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1887240640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f18800d1190 con 0x7f18800ceb00 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1887240640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f18800b5530 con 0x7f1880161760 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18800ceb00 0x7f18800d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1884fdc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 0x7f18800b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18800ceb00 0x7f18800d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33894/0 (socket says 172.21.15.1:33894) 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f18857dd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f18800d0d00 0x7f1880161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:55460/0 (socket says 172.21.15.1:55460) 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 -- 172.21.15.1:0/504710546 learned_addr learned my addr 172.21.15.1:0/504710546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 -- 172.21.15.1:0/504710546 >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 msgr2=0x7f18800b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 --2- 172.21.15.1:0/504710546 >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 0x7f18800b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 -- 172.21.15.1:0/504710546 >> v1:172.21.15.3:6789/0 conn(0x7f18800d0d00 legacy=0x7f1880161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f187ffff640 1 -- 172.21.15.1:0/504710546 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18800b6d40 con 0x7f18800ceb00 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.172+0000 7f1884fdc640 1 --2- 172.21.15.1:0/504710546 >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 0x7f18800b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f187ffff640 1 --2- 172.21.15.1:0/504710546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18800d07c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f188008c9f0 tx=0x7f186c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e512af8af2f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f187f7fe640 1 -- 172.21.15.1:0/504710546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c031870 con 0x7f18800ceb00 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f187f7fe640 1 -- 172.21.15.1:0/504710546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186c0319d0 con 0x7f18800ceb00 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f187f7fe640 1 -- 172.21.15.1:0/504710546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c031d60 con 0x7f18800ceb00 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f1887240640 1 -- 172.21.15.1:0/504710546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 msgr2=0x7f18800d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.173+0000 7f1887240640 1 --2- 172.21.15.1:0/504710546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18800d07c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f188008c9f0 tx=0x7f186c02b910 comp rx=0 tx=0).stop 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 -- 172.21.15.1:0/504710546 shutdown_connections 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 --2- 172.21.15.1:0/504710546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18800d07c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 --2- 172.21.15.1:0/504710546 >> v2:172.21.15.3:3300/0 conn(0x7f1880161760 0x7f18800b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 -- 172.21.15.1:0/504710546 >> 172.21.15.1:0/504710546 conn(0x7f188009d1c0 msgr2=0x7f18800d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 -- 172.21.15.1:0/504710546 shutdown_connections 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.174+0000 7f1887240640 1 -- 172.21.15.1:0/504710546 wait complete. 2024-03-24T10:34:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.175+0000 7f1887240640 1 Processor -- start 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.175+0000 7f1887240640 1 -- start start 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18800d0d00 0x7f18800c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1880161760 0x7f18800c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1880160730 con 0x7f18800ceb00 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f18800cc180 con 0x7f18800d0d00 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1887240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f18800cc460 con 0x7f1880161760 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33900/0 (socket says 172.21.15.1:33900) 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 -- 172.21.15.1:0/364015489 learned_addr learned my addr 172.21.15.1:0/364015489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f187ffff640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18800d0d00 0x7f18800c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f18857dd640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1880161760 0x7f18800c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 -- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1880161760 msgr2=0x7f18800c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1880161760 0x7f18800c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 -- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18800d0d00 msgr2=0x7f18800c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18800d0d00 0x7f18800c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 -- 172.21.15.1:0/364015489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1874009400 con 0x7f18800ceb00 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.176+0000 7f1884fdc640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f187400e960 tx=0x7f187400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.177+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187400ca00 con 0x7f18800ceb00 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.177+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187400cb60 con 0x7f18800ceb00 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.177+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186c0094d0 con 0x7f18800ceb00 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.177+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874013470 con 0x7f18800ceb00 2024-03-24T10:34:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.177+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18800cca20 con 0x7f18800ceb00 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.178+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188004f4e0 con 0x7f18800ceb00 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.178+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f18740135d0 con 0x7f18800ceb00 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.179+0000 7f187d7fa640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 0x7f186406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.179+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f187408a360 con 0x7f18800ceb00 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.179+0000 7f187ffff640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 0x7f186406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.179+0000 7f187ffff640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 0x7f186406dd00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f186c036000 tx=0x7f186c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.183+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1874054e30 con 0x7f18800ceb00 2024-03-24T10:34:06.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.363+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18800d21b0 con 0x7f18800ceb00 2024-03-24T10:34:06.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.363+0000 7f187d7fa640 1 -- 172.21.15.1:0/364015489 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f18740584f0 con 0x7f18800ceb00 2024-03-24T10:34:06.363 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:06.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 msgr2=0x7f186406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 0x7f186406dd00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f186c036000 tx=0x7f186c03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:06.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 msgr2=0x7f18801511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f187400e960 tx=0x7f187400ee20 comp rx=0 tx=0).stop 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 shutdown_connections 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b840 0x7f186406dd00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1880161760 0x7f18800c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18800d0d00 0x7f18800c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 --2- 172.21.15.1:0/364015489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18800ceb00 0x7f18801511a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 >> 172.21.15.1:0/364015489 conn(0x7f188009d1c0 msgr2=0x7f18800b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 shutdown_connections 2024-03-24T10:34:06.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.365+0000 7f1887240640 1 -- 172.21.15.1:0/364015489 wait complete. 2024-03-24T10:34:06.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.482+0000 7f651372e640 1 Processor -- start 2024-03-24T10:34:06.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.482+0000 7f651372e640 1 -- start start 2024-03-24T10:34:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c15a4d0 0x7f650c06c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f650c0b4ca0 0x7f650c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f650c0cc270 con 0x7f650c06d7a0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f650c0b4230 con 0x7f650c0b4ca0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f651372e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f650c06ce70 con 0x7f650c15a4d0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6511ccb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f650c0b4ca0 0x7f650c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f65114ca640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c15a4d0 0x7f650c06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f650c0b4ca0 0x7f650c06d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:50056/0 (socket says 172.21.15.1:50056) 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 -- 172.21.15.1:0/1133331949 learned_addr learned my addr 172.21.15.1:0/1133331949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6511ccb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33910/0 (socket says 172.21.15.1:33910) 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 -- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c15a4d0 msgr2=0x7f650c06c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 --2- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c15a4d0 0x7f650c06c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 -- 172.21.15.1:0/1133331949 >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 msgr2=0x7f650c15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 --2- 172.21.15.1:0/1133331949 >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6510cc9640 1 -- 172.21.15.1:0/1133331949 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650c06dc00 con 0x7f650c0b4ca0 2024-03-24T10:34:06.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.483+0000 7f6511ccb640 1 --2- 172.21.15.1:0/1133331949 >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.484+0000 7f6510cc9640 1 --2- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c0b4ca0 0x7f650c06d260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f650c0d09d0 tx=0x7f64fc02b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e717926dde2275 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.484+0000 7f6503fff640 1 -- 172.21.15.1:0/1133331949 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc031860 con 0x7f650c0b4ca0 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.484+0000 7f6503fff640 1 -- 172.21.15.1:0/1133331949 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc0319c0 con 0x7f650c0b4ca0 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.484+0000 7f6503fff640 1 -- 172.21.15.1:0/1133331949 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc031d50 con 0x7f650c0b4ca0 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 -- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c0b4ca0 msgr2=0x7f650c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 --2- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c0b4ca0 0x7f650c06d260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f650c0d09d0 tx=0x7f64fc02b900 comp rx=0 tx=0).stop 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 -- 172.21.15.1:0/1133331949 shutdown_connections 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 --2- 172.21.15.1:0/1133331949 >> v2:172.21.15.1:3300/0 conn(0x7f650c06d7a0 0x7f650c15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 --2- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c0b4ca0 0x7f650c06d260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 --2- 172.21.15.1:0/1133331949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c15a4d0 0x7f650c06c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 -- 172.21.15.1:0/1133331949 >> 172.21.15.1:0/1133331949 conn(0x7f650c06b4f0 msgr2=0x7f650c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 -- 172.21.15.1:0/1133331949 shutdown_connections 2024-03-24T10:34:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.485+0000 7f651372e640 1 -- 172.21.15.1:0/1133331949 wait complete. 2024-03-24T10:34:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.486+0000 7f651372e640 1 Processor -- start 2024-03-24T10:34:06.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.486+0000 7f651372e640 1 -- start start 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c06d7a0 0x7f650c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c15a4d0 0x7f650c1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f650c1418c0 con 0x7f650c0b4ca0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f650c13e8e0 con 0x7f650c15a4d0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f650c13eb80 con 0x7f650c06d7a0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33912/0 (socket says 172.21.15.1:33912) 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 -- 172.21.15.1:0/2504849468 learned_addr learned my addr 172.21.15.1:0/2504849468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f65114ca640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c06d7a0 0x7f650c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6511ccb640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c15a4d0 0x7f650c1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 -- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c06d7a0 msgr2=0x7f650c141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c06d7a0 0x7f650c141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 -- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c15a4d0 msgr2=0x7f650c1576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c15a4d0 0x7f650c1576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 -- 172.21.15.1:0/2504849468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6504009a70 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6510cc9640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64fc02b940 tx=0x7f64fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc03ac60 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc03adc0 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64fc0094d0 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.487+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc044630 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.488+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650c13f140 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.488+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f650c04f470 con 0x7f650c0b4ca0 2024-03-24T10:34:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.489+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f64fc04d450 con 0x7f650c0b4ca0 2024-03-24T10:34:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.489+0000 7f6501ffb640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 0x7f64f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.489+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f64fc0ac4b0 con 0x7f650c0b4ca0 2024-03-24T10:34:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.489+0000 7f65114ca640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 0x7f64f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.490+0000 7f65114ca640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 0x7f64f006dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6508009aa0 tx=0x7f6508009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.494+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f64fc076f60 con 0x7f650c0b4ca0 2024-03-24T10:34:06.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.675+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f650c0b3310 con 0x7f650c0b4ca0 2024-03-24T10:34:06.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.676+0000 7f6501ffb640 1 -- 172.21.15.1:0/2504849468 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f64fc07a620 con 0x7f650c0b4ca0 2024-03-24T10:34:06.677 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:06.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 msgr2=0x7f64f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 0x7f64f006dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6508009aa0 tx=0x7f6508009190 comp rx=0 tx=0).stop 2024-03-24T10:34:06.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 msgr2=0x7f650c13e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64fc02b940 tx=0x7f64fc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 shutdown_connections 2024-03-24T10:34:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64f006b810 0x7f64f006dcd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f650c15a4d0 0x7f650c1576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f650c0b4ca0 0x7f650c13e010 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 --2- 172.21.15.1:0/2504849468 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f650c06d7a0 0x7f650c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 >> 172.21.15.1:0/2504849468 conn(0x7f650c06b4f0 msgr2=0x7f650c0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 shutdown_connections 2024-03-24T10:34:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.678+0000 7f651372e640 1 -- 172.21.15.1:0/2504849468 wait complete. 2024-03-24T10:34:06.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.798+0000 7fed72313640 1 Processor -- start 2024-03-24T10:34:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.798+0000 7fed72313640 1 -- start start 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed72313640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fed6c1600e0 0x7fed6c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed72313640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c145b10 con 0x7fed6c145730 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed72313640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c071be0 con 0x7fed6c06d120 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed72313640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fed6c06d620 con 0x7fed6c1600e0 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fed6c145730 0x7fed6c15f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41696/0 (socket says 172.21.15.1:41696) 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6bfff640 1 -- 172.21.15.1:0/3103232898 learned_addr learned my addr 172.21.15.1:0/3103232898 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6b7fe640 1 --2- 172.21.15.1:0/3103232898 >> v2:172.21.15.3:3300/0 conn(0x7fed6c1600e0 0x7fed6c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6affd640 1 -- 172.21.15.1:0/3103232898 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877261567 0 0) 0x7fed6c145b10 con 0x7fed6c145730 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6affd640 1 -- 172.21.15.1:0/3103232898 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed54003520 con 0x7fed6c145730 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6b7fe640 1 -- 172.21.15.1:0/3103232898 >> v1:172.21.15.3:6789/0 conn(0x7fed6c06d120 legacy=0x7fed6c0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6b7fe640 1 -- 172.21.15.1:0/3103232898 >> v1:172.21.15.1:6789/0 conn(0x7fed6c145730 legacy=0x7fed6c15f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.799+0000 7fed6b7fe640 1 -- 172.21.15.1:0/3103232898 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c06d900 con 0x7fed6c1600e0 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.800+0000 7fed6b7fe640 1 --2- 172.21.15.1:0/3103232898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c0ce260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0097a0 tx=0x7fed5c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e2354ef4ab1d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.800+0000 7fed6affd640 1 -- 172.21.15.1:0/3103232898 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c030610 con 0x7fed6c1600e0 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.800+0000 7fed6affd640 1 -- 172.21.15.1:0/3103232898 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5c030770 con 0x7fed6c1600e0 2024-03-24T10:34:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.800+0000 7fed6affd640 1 -- 172.21.15.1:0/3103232898 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c030ae0 con 0x7fed6c1600e0 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 -- 172.21.15.1:0/3103232898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 msgr2=0x7fed6c0ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 --2- 172.21.15.1:0/3103232898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c0ce260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0097a0 tx=0x7fed5c02b960 comp rx=0 tx=0).stop 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 -- 172.21.15.1:0/3103232898 shutdown_connections 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 --2- 172.21.15.1:0/3103232898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c0ce260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 -- 172.21.15.1:0/3103232898 >> 172.21.15.1:0/3103232898 conn(0x7fed6c0b1710 msgr2=0x7fed6c0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 -- 172.21.15.1:0/3103232898 shutdown_connections 2024-03-24T10:34:06.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.801+0000 7fed72313640 1 -- 172.21.15.1:0/3103232898 wait complete. 2024-03-24T10:34:06.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.802+0000 7fed72313640 1 Processor -- start 2024-03-24T10:34:06.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.802+0000 7fed72313640 1 -- start start 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fed6c145730 0x7fed6c15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c13c350 con 0x7fed6c06d120 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c13b260 con 0x7fed6c1600e0 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c13b540 con 0x7fed6c145730 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33920/0 (socket says 172.21.15.1:33920) 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 -- 172.21.15.1:0/3695417924 learned_addr learned my addr 172.21.15.1:0/3695417924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6b7fe640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fed6c145730 0x7fed6c15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed708b0640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 -- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fed6c145730 msgr2=0x7fed6c15c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fed6c145730 0x7fed6c15c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 -- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 msgr2=0x7fed6c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c13f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 -- 172.21.15.1:0/3695417924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c009400 con 0x7fed6c06d120 2024-03-24T10:34:06.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed6bfff640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed6000c500 tx=0x7fed6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60005ce0 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed60005e40 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c13b820 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.803+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60007330 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.804+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c15b7f0 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.804+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6c13c4c0 con 0x7fed6c06d120 2024-03-24T10:34:06.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.805+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fed60007490 con 0x7fed6c06d120 2024-03-24T10:34:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.805+0000 7fed68ff9640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 0x7fed5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.805+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fed60089800 con 0x7fed6c06d120 2024-03-24T10:34:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.805+0000 7fed6b7fe640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 0x7fed5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:06.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.806+0000 7fed6b7fe640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 0x7fed5806dd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fed5c036000 tx=0x7fed5c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:06.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.810+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fed6c13c4c0 con 0x7fed6c06d120 2024-03-24T10:34:06.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.993+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed6c13c490 con 0x7fed6c06d120 2024-03-24T10:34:06.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.994+0000 7fed68ff9640 1 -- 172.21.15.1:0/3695417924 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fed600170a0 con 0x7fed6c06d120 2024-03-24T10:34:06.994 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 msgr2=0x7fed5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 0x7fed5806dd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fed5c036000 tx=0x7fed5c008030 comp rx=0 tx=0).stop 2024-03-24T10:34:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 msgr2=0x7fed6c15b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed6000c500 tx=0x7fed6000c9c0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 shutdown_connections 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fed5806b840 0x7fed5806dd00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c1600e0 0x7fed6c13f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fed6c145730 0x7fed6c15c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 --2- 172.21.15.1:0/3695417924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed6c06d120 0x7fed6c15b2b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 >> 172.21.15.1:0/3695417924 conn(0x7fed6c0b1710 msgr2=0x7fed6c144590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 shutdown_connections 2024-03-24T10:34:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:06.996+0000 7fed72313640 1 -- 172.21.15.1:0/3695417924 wait complete. 2024-03-24T10:34:07.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.109+0000 7f9b21b63640 1 Processor -- start 2024-03-24T10:34:07.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.109+0000 7f9b21b63640 1 -- start start 2024-03-24T10:34:07.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b21b63640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b1c144770 0x7f9b1c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b21b63640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b1c070b90 0x7f9b1c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b21b63640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1c0cdd60 con 0x7f9b1c0b2aa0 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b21b63640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9b1c06eaa0 con 0x7f9b1c144770 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b21b63640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9b1c071040 con 0x7f9b1c070b90 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b1c144770 0x7f9b1c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b1c144770 0x7f9b1c0d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33922/0 (socket says 172.21.15.1:33922) 2024-03-24T10:34:07.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/642399144 learned_addr learned my addr 172.21.15.1:0/642399144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1affd640 1 --2- 172.21.15.1:0/642399144 >> v2:172.21.15.3:3300/0 conn(0x7f9b1c070b90 0x7f9b1c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/642399144 >> v2:172.21.15.3:3300/0 conn(0x7f9b1c070b90 msgr2=0x7f9b1c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 --2- 172.21.15.1:0/642399144 >> v2:172.21.15.3:3300/0 conn(0x7f9b1c070b90 0x7f9b1c06da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/642399144 >> v1:172.21.15.3:6789/0 conn(0x7f9b1c0b2aa0 legacy=0x7f9b1c0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.110+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/642399144 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c06df60 con 0x7f9b1c144770 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.111+0000 7f9b1b7fe640 1 --2- 172.21.15.1:0/642399144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c144770 0x7f9b1c0d00f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cf0f0 tx=0x7f9b0c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242c0d967fb09968 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.111+0000 7f9b1a7fc640 1 -- 172.21.15.1:0/642399144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c030860 con 0x7f9b1c144770 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.111+0000 7f9b1a7fc640 1 -- 172.21.15.1:0/642399144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0c0309c0 con 0x7f9b1c144770 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.111+0000 7f9b1a7fc640 1 -- 172.21.15.1:0/642399144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c030d50 con 0x7f9b1c144770 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 -- 172.21.15.1:0/642399144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c144770 msgr2=0x7f9b1c0d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 --2- 172.21.15.1:0/642399144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c144770 0x7f9b1c0d00f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0cf0f0 tx=0x7f9b0c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 -- 172.21.15.1:0/642399144 shutdown_connections 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 --2- 172.21.15.1:0/642399144 >> v2:172.21.15.3:3300/0 conn(0x7f9b1c070b90 0x7f9b1c06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 --2- 172.21.15.1:0/642399144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c144770 0x7f9b1c0d00f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 -- 172.21.15.1:0/642399144 >> 172.21.15.1:0/642399144 conn(0x7f9b1c0b17a0 msgr2=0x7f9b1c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 -- 172.21.15.1:0/642399144 shutdown_connections 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.112+0000 7f9b21b63640 1 -- 172.21.15.1:0/642399144 wait complete. 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.113+0000 7f9b21b63640 1 Processor -- start 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.113+0000 7f9b21b63640 1 -- start start 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 0x7f9b1c0c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b1c144770 0x7f9b1c0c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c134900 con 0x7f9b1c070b90 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c134be0 con 0x7f9b1c0b2aa0 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b21b63640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c134ec0 con 0x7f9b1c144770 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33930/0 (socket says 172.21.15.1:33930) 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/3439107101 learned_addr learned my addr 172.21.15.1:0/3439107101 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1affd640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 0x7f9b1c0c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1bfff640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b1c144770 0x7f9b1c0c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b1c144770 msgr2=0x7f9b1c0c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b1c144770 0x7f9b1c0c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 msgr2=0x7f9b1c0c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 0x7f9b1c0c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 -- 172.21.15.1:0/3439107101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c0094d0 con 0x7f9b1c070b90 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1affd640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 0x7f9b1c0c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.114+0000 7f9b1b7fe640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c02b9e0 tx=0x7f9b0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.115+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c039a90 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.115+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0c039c10 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.115+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c043410 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.115+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c1351a0 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.115+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c134520 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.116+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1c1356c0 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.116+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9b0c04c3b0 con 0x7f9b1c070b90 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.116+0000 7f9b03fff640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 0x7f9b0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.116+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9b0c0ac510 con 0x7f9b1c070b90 2024-03-24T10:34:07.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.116+0000 7f9b1affd640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 0x7f9b0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.117+0000 7f9b1affd640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 0x7f9b0806dcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009f30 tx=0x7f9b10009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.121+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9b1c1356c0 con 0x7f9b1c070b90 2024-03-24T10:34:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.344+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b1c1356c0 con 0x7f9b1c070b90 2024-03-24T10:34:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.344+0000 7f9b03fff640 1 -- 172.21.15.1:0/3439107101 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f9b1c1356c0 con 0x7f9b1c070b90 2024-03-24T10:34:07.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:07.345 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 msgr2=0x7f9b0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 0x7f9b0806dcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009f30 tx=0x7f9b10009190 comp rx=0 tx=0).stop 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 msgr2=0x7f9b1c0c4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c02b9e0 tx=0x7f9b0c004730 comp rx=0 tx=0).stop 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 shutdown_connections 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b0806b810 0x7f9b0806dcd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b1c144770 0x7f9b1c0c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b1c0b2aa0 0x7f9b1c0c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 --2- 172.21.15.1:0/3439107101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b1c070b90 0x7f9b1c0c4d20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.346+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 >> 172.21.15.1:0/3439107101 conn(0x7f9b1c0b17a0 msgr2=0x7f9b1c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.347+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 shutdown_connections 2024-03-24T10:34:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.347+0000 7f9b21b63640 1 -- 172.21.15.1:0/3439107101 wait complete. 2024-03-24T10:34:07.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.462+0000 7ff7ef75c640 1 Processor -- start 2024-03-24T10:34:07.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.462+0000 7ff7ef75c640 1 -- start start 2024-03-24T10:34:07.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ef75c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7e80b7530 0x7ff7e80d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ef75c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ef75c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e806d520 con 0x7ff7e8161c60 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ef75c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff7e80d1320 con 0x7ff7e80b7530 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ef75c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e80b4c70 con 0x7ff7e806cba0 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7e80b7530 0x7ff7e80d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7e80b7530 0x7ff7e80d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33934/0 (socket says 172.21.15.1:33934) 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 -- 172.21.15.1:0/104284108 learned_addr learned my addr 172.21.15.1:0/104284108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7eccf7640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 -- 172.21.15.1:0/104284108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 msgr2=0x7ff7e806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 -- 172.21.15.1:0/104284108 >> v1:172.21.15.3:6789/0 conn(0x7ff7e8161c60 legacy=0x7ff7e81620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.463+0000 7ff7ed4f8640 1 -- 172.21.15.1:0/104284108 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e80b4f50 con 0x7ff7e80b7530 2024-03-24T10:34:07.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.464+0000 7ff7ed4f8640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e80b7530 0x7ff7e80d09b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009e50 tx=0x7ff7d802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c5dd399f66b928 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.464+0000 7ff7e7fff640 1 -- 172.21.15.1:0/104284108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d802cb00 con 0x7ff7e80b7530 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.464+0000 7ff7e7fff640 1 -- 172.21.15.1:0/104284108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d802cc60 con 0x7ff7e80b7530 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.464+0000 7ff7e7fff640 1 -- 172.21.15.1:0/104284108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d80343f0 con 0x7ff7e80b7530 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 -- 172.21.15.1:0/104284108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e80b7530 msgr2=0x7ff7e80d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e80b7530 0x7ff7e80d09b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009e50 tx=0x7ff7d802bf60 comp rx=0 tx=0).stop 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 -- 172.21.15.1:0/104284108 shutdown_connections 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/104284108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e80b7530 0x7ff7e80d09b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 -- 172.21.15.1:0/104284108 >> 172.21.15.1:0/104284108 conn(0x7ff7e80d1c80 msgr2=0x7ff7e80d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 -- 172.21.15.1:0/104284108 shutdown_connections 2024-03-24T10:34:07.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.465+0000 7ff7ef75c640 1 -- 172.21.15.1:0/104284108 wait complete. 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.466+0000 7ff7ef75c640 1 Processor -- start 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.466+0000 7ff7ef75c640 1 -- start start 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e80cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7e80b7530 0x7ff7e81f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e80ca530 con 0x7ff7e8161c60 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e80ca780 con 0x7ff7e80b7530 2024-03-24T10:34:07.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e80caa60 con 0x7ff7e806cba0 2024-03-24T10:34:07.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ed4f8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e80cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33944/0 (socket says 172.21.15.1:33944) 2024-03-24T10:34:07.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7eccf7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7e80b7530 0x7ff7e81f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 -- 172.21.15.1:0/1007402674 learned_addr learned my addr 172.21.15.1:0/1007402674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 -- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 msgr2=0x7ff7e80cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e80cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 -- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7e80b7530 msgr2=0x7ff7e81f77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7e80b7530 0x7ff7e81f77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 -- 172.21.15.1:0/1007402674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d8009b30 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7edcf9640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0001eb0 tx=0x7ff7e000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00052d0 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e0005430 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e80cad40 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.467+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0003810 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.468+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e80cb300 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.469+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e806de20 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.469+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff7e0003970 con 0x7ff7e8161c60 2024-03-24T10:34:07.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.469+0000 7ff7e5ffb640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 0x7ff7d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.469+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff7e0089660 con 0x7ff7e8161c60 2024-03-24T10:34:07.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.469+0000 7ff7ed4f8640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 0x7ff7d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.470+0000 7ff7ed4f8640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 0x7ff7d406dcd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8035000 tx=0x7ff7d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.474+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff7e00541c0 con 0x7ff7e8161c60 2024-03-24T10:34:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.655+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7e804f4e0 con 0x7ff7e8161c60 2024-03-24T10:34:07.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.656+0000 7ff7e5ffb640 1 -- 172.21.15.1:0/1007402674 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7ff7e0057880 con 0x7ff7e8161c60 2024-03-24T10:34:07.656 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:07.657 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 msgr2=0x7ff7d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 0x7ff7d406dcd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8035000 tx=0x7ff7d8030000 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 msgr2=0x7ff7e80c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0001eb0 tx=0x7ff7e000c890 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 shutdown_connections 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff7d406b810 0x7ff7d406dcd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7e8161c60 0x7ff7e80c9780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7e80b7530 0x7ff7e81f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 --2- 172.21.15.1:0/1007402674 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff7e806cba0 0x7ff7e80cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 >> 172.21.15.1:0/1007402674 conn(0x7ff7e80d1c80 msgr2=0x7ff7e80ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 shutdown_connections 2024-03-24T10:34:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.658+0000 7ff7ef75c640 1 -- 172.21.15.1:0/1007402674 wait complete. 2024-03-24T10:34:07.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:07.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:07.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.775+0000 7f13a9981640 1 Processor -- start 2024-03-24T10:34:07.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.775+0000 7f13a9981640 1 -- start start 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13a4158540 0x7f13a406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13a406b5e0 0x7f13a40af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a40aff30 0x7f13a40ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f13a406ba60 con 0x7f13a406b5e0 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f13a40b03c0 con 0x7f13a4158540 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a9981640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f13a4072760 con 0x7f13a40aff30 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13a406b5e0 0x7f13a40af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a2ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13a4158540 0x7f13a406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13a406b5e0 0x7f13a40af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33954/0 (socket says 172.21.15.1:33954) 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 -- 172.21.15.1:0/2441087471 learned_addr learned my addr 172.21.15.1:0/2441087471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f13a37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a40aff30 0x7f13a40ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 -- 172.21.15.1:0/2441087471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a40aff30 msgr2=0x7f13a40ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 --2- 172.21.15.1:0/2441087471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a40aff30 0x7f13a40ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 -- 172.21.15.1:0/2441087471 >> v2:172.21.15.3:3300/0 conn(0x7f13a4158540 msgr2=0x7f13a406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 --2- 172.21.15.1:0/2441087471 >> v2:172.21.15.3:3300/0 conn(0x7f13a4158540 0x7f13a406b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.776+0000 7f139bfff640 1 -- 172.21.15.1:0/2441087471 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a40b05d0 con 0x7f13a406b5e0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f139bfff640 1 --2- 172.21.15.1:0/2441087471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a406b5e0 0x7f13a40af9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13a40731a0 tx=0x7f138c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b23eb9afa2cdeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a27fc640 1 -- 172.21.15.1:0/2441087471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0317e0 con 0x7f13a406b5e0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a27fc640 1 -- 172.21.15.1:0/2441087471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c031940 con 0x7f13a406b5e0 2024-03-24T10:34:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a27fc640 1 -- 172.21.15.1:0/2441087471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c031cd0 con 0x7f13a406b5e0 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a9981640 1 -- 172.21.15.1:0/2441087471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a406b5e0 msgr2=0x7f13a40af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a9981640 1 --2- 172.21.15.1:0/2441087471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a406b5e0 0x7f13a40af9f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13a40731a0 tx=0x7f138c02b880 comp rx=0 tx=0).stop 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.777+0000 7f13a9981640 1 -- 172.21.15.1:0/2441087471 shutdown_connections 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 --2- 172.21.15.1:0/2441087471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a40aff30 0x7f13a40ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 --2- 172.21.15.1:0/2441087471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a406b5e0 0x7f13a40af9f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 --2- 172.21.15.1:0/2441087471 >> v2:172.21.15.3:3300/0 conn(0x7f13a4158540 0x7f13a406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 -- 172.21.15.1:0/2441087471 >> 172.21.15.1:0/2441087471 conn(0x7f13a40b2820 msgr2=0x7f13a40b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 -- 172.21.15.1:0/2441087471 shutdown_connections 2024-03-24T10:34:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.778+0000 7f13a9981640 1 -- 172.21.15.1:0/2441087471 wait complete. 2024-03-24T10:34:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 Processor -- start 2024-03-24T10:34:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 -- start start 2024-03-24T10:34:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a406b5e0 0x7f13a414d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13a4158540 0x7f13a40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41f7f60 con 0x7f13a40aff30 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f13a414dd90 con 0x7f13a4158540 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.779+0000 7f13a9981640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f13a41f7390 con 0x7f13a406b5e0 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33970/0 (socket says 172.21.15.1:33970) 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 -- 172.21.15.1:0/2308673346 learned_addr learned my addr 172.21.15.1:0/2308673346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a2ffd640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a406b5e0 0x7f13a414d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a37fe640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13a4158540 0x7f13a40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 -- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a406b5e0 msgr2=0x7f13a414d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a406b5e0 0x7f13a414d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 -- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13a4158540 msgr2=0x7f13a40d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13a4158540 0x7f13a40d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 -- 172.21.15.1:0/2308673346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138c0094d0 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f139bfff640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f138c000c00 tx=0x7f138c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c032880 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c032a00 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a41f7630 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c032db0 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.780+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a41f7bf0 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.781+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a41f7d80 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.782+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f138c0453f0 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.782+0000 7f13a0ff9640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 0x7f138406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.782+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f138c0ac1d0 con 0x7f13a40aff30 2024-03-24T10:34:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.782+0000 7f13a2ffd640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 0x7f138406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.782+0000 7f13a2ffd640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 0x7f138406dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1394009b40 tx=0x7f1394009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.786+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f138c076d30 con 0x7f13a40aff30 2024-03-24T10:34:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.966+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13a41f8110 con 0x7f13a40aff30 2024-03-24T10:34:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.966+0000 7f13a0ff9640 1 -- 172.21.15.1:0/2308673346 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f138c07a3f0 con 0x7f13a40aff30 2024-03-24T10:34:07.967 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 msgr2=0x7f138406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 0x7f138406dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1394009b40 tx=0x7f1394009190 comp rx=0 tx=0).stop 2024-03-24T10:34:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 msgr2=0x7f13a4152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f138c000c00 tx=0x7f138c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 shutdown_connections 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f138406b810 0x7f138406dcd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13a4158540 0x7f13a40d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a40aff30 0x7f13a4152f30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 --2- 172.21.15.1:0/2308673346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f13a406b5e0 0x7f13a414d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 >> 172.21.15.1:0/2308673346 conn(0x7f13a40b2820 msgr2=0x7f13a406dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 shutdown_connections 2024-03-24T10:34:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:07.968+0000 7f13a9981640 1 -- 172.21.15.1:0/2308673346 wait complete. 2024-03-24T10:34:07.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:08.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 Processor -- start 2024-03-24T10:34:08.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 -- start start 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf415e340 0x7f5cf40b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cf40d0dd0 0x7f5cf40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf40b0d40 con 0x7f5cf40d22e0 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf40d48c0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.086+0000 7f5cf9202640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf40d4ba0 con 0x7f5cf415e340 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cf40d0dd0 0x7f5cf40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cf40d0dd0 0x7f5cf40b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33982/0 (socket says 172.21.15.1:33982) 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 -- 172.21.15.1:0/3200877930 learned_addr learned my addr 172.21.15.1:0/3200877930 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cebfff640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf415e340 0x7f5cf40b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 -- 172.21.15.1:0/3200877930 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf415e340 msgr2=0x7f5cf40b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf415e340 0x7f5cf40b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 -- 172.21.15.1:0/3200877930 >> v1:172.21.15.3:6789/0 conn(0x7f5cf40d22e0 legacy=0x7f5cf40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.087+0000 7f5cf359e640 1 -- 172.21.15.1:0/3200877930 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf40b09d0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf359e640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf40b1710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009e50 tx=0x7f5cec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9a87ee643e1939 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf259c640 1 -- 172.21.15.1:0/3200877930 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec02cb00 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf259c640 1 -- 172.21.15.1:0/3200877930 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cec02cc60 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf259c640 1 -- 172.21.15.1:0/3200877930 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0343f0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 -- 172.21.15.1:0/3200877930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 msgr2=0x7f5cf40b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf40b1710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009e50 tx=0x7f5cec02bf60 comp rx=0 tx=0).stop 2024-03-24T10:34:08.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 -- 172.21.15.1:0/3200877930 shutdown_connections 2024-03-24T10:34:08.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf40b1710 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 --2- 172.21.15.1:0/3200877930 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf415e340 0x7f5cf40b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 -- 172.21.15.1:0/3200877930 >> 172.21.15.1:0/3200877930 conn(0x7f5cf40d3d00 msgr2=0x7f5cf40d4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 -- 172.21.15.1:0/3200877930 shutdown_connections 2024-03-24T10:34:08.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.088+0000 7f5cf9202640 1 -- 172.21.15.1:0/3200877930 wait complete. 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 Processor -- start 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 -- start start 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 0x7f5cf40b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 0x7f5cf40b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4155460 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4160360 con 0x7f5cf415e340 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf9202640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf415e750 con 0x7f5cf40d22e0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.090+0000 7f5cf2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33992/0 (socket says 172.21.15.1:33992) 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 -- 172.21.15.1:0/200597240 learned_addr learned my addr 172.21.15.1:0/200597240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cebfff640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 0x7f5cf40b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf359e640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 0x7f5cf40b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 -- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 msgr2=0x7f5cf40b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 0x7f5cf40b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 -- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 msgr2=0x7f5cf40b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 0x7f5cf40b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 -- 172.21.15.1:0/200597240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cec009b30 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf359e640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 0x7f5cf40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cebfff640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 0x7f5cf40b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf2d9d640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400c400 tx=0x7f5ce400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40052d0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce4005450 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf415ea30 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4007260 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.091+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf415eff0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.092+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf40ce480 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.093+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ce40073c0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.093+0000 7f5ceb7fe640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 0x7f5cd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.093+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ce4089540 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.093+0000 7f5cebfff640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 0x7f5cd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.093+0000 7f5cebfff640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 0x7f5cd406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4160660 tx=0x7f5cdc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.097+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ce4054010 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.277+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5cf404f470 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.277+0000 7f5ceb7fe640 1 -- 172.21.15.1:0/200597240 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f5ce40576d0 con 0x7f5cf40d0dd0 2024-03-24T10:34:08.278 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 msgr2=0x7f5cd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 0x7f5cd406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4160660 tx=0x7f5cdc009190 comp rx=0 tx=0).stop 2024-03-24T10:34:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 msgr2=0x7f5cf414c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400c400 tx=0x7f5ce400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 shutdown_connections 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5cd406b810 0x7f5cd406dcd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cf415e340 0x7f5cf40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5cf40d22e0 0x7f5cf40b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 --2- 172.21.15.1:0/200597240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d0dd0 0x7f5cf414c2b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 >> 172.21.15.1:0/200597240 conn(0x7f5cf40d3d00 msgr2=0x7f5cf406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 shutdown_connections 2024-03-24T10:34:08.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.279+0000 7f5cf9202640 1 -- 172.21.15.1:0/200597240 wait complete. 2024-03-24T10:34:08.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:08.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.394+0000 7fbb8a623640 1 Processor -- start 2024-03-24T10:34:08.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.394+0000 7fbb8a623640 1 -- start start 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb8a623640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 0x7fbb8406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb8a623640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb840cc2b0 con 0x7fbb8406f780 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb8a623640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb840cc870 con 0x7fbb840cdd10 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb8a623640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb840cf460 con 0x7fbb8415e2c0 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb837fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb8406f780 0x7fbb840ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41708/0 (socket says 172.21.15.1:41708) 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb837fe640 1 -- 172.21.15.1:0/4189471000 learned_addr learned my addr 172.21.15.1:0/4189471000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb83fff640 1 --2- 172.21.15.1:0/4189471000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 0x7fbb8406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869430235 0 0) 0x7fbb840cc2b0 con 0x7fbb8406f780 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb70003520 con 0x7fbb8406f780 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392396687 0 0) 0x7fbb840cc870 con 0x7fbb840cdd10 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb83fff640 1 -- 172.21.15.1:0/4189471000 >> v1:172.21.15.3:6789/0 conn(0x7fbb840cdd10 legacy=0x7fbb840ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb83fff640 1 -- 172.21.15.1:0/4189471000 >> v1:172.21.15.1:6789/0 conn(0x7fbb8406f780 legacy=0x7fbb840ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.395+0000 7fbb83fff640 1 -- 172.21.15.1:0/4189471000 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb840d0d50 con 0x7fbb8415e2c0 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.396+0000 7fbb83fff640 1 --2- 172.21.15.1:0/4189471000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 0x7fbb8406f130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097d0 tx=0x7fbb7402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f255e5800149ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.396+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74031800 con 0x7fbb8415e2c0 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.396+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb74031960 con 0x7fbb8415e2c0 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.396+0000 7fbb82ffd640 1 -- 172.21.15.1:0/4189471000 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74031cf0 con 0x7fbb8415e2c0 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 -- 172.21.15.1:0/4189471000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 msgr2=0x7fbb8406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 --2- 172.21.15.1:0/4189471000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 0x7fbb8406f130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097d0 tx=0x7fbb7402b8a0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 -- 172.21.15.1:0/4189471000 shutdown_connections 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 --2- 172.21.15.1:0/4189471000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb8415e2c0 0x7fbb8406f130 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 -- 172.21.15.1:0/4189471000 >> 172.21.15.1:0/4189471000 conn(0x7fbb840d1c60 msgr2=0x7fbb840d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 -- 172.21.15.1:0/4189471000 shutdown_connections 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.397+0000 7fbb8a623640 1 -- 172.21.15.1:0/4189471000 wait complete. 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.398+0000 7fbb8a623640 1 Processor -- start 2024-03-24T10:34:08.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.398+0000 7fbb8a623640 1 -- start start 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb840cdd10 0x7fbb840c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb8415e2c0 0x7fbb8415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb840b6d60 con 0x7fbb8406f780 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbb840c9390 con 0x7fbb8415e2c0 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb8a623640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb840b60e0 con 0x7fbb840cdd10 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33996/0 (socket says 172.21.15.1:33996) 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 -- 172.21.15.1:0/3778125459 learned_addr learned my addr 172.21.15.1:0/3778125459 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb837fe640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb840cdd10 0x7fbb840c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb88bc0640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb8415e2c0 0x7fbb8415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 -- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb840cdd10 msgr2=0x7fbb840c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb840cdd10 0x7fbb840c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 -- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb8415e2c0 msgr2=0x7fbb8415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb8415e2c0 0x7fbb8415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 -- 172.21.15.1:0/3778125459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb74009400 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.399+0000 7fbb83fff640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097d0 tx=0x7fbb740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.400+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74004230 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.400+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb74004390 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.400+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb840b63c0 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.400+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7403baa0 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.400+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb840b6980 con 0x7fbb8406f780 2024-03-24T10:34:08.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.401+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb840b6ea0 con 0x7fbb8406f780 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.401+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb74043070 con 0x7fbb8406f780 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.401+0000 7fbb80ff9640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 0x7fbb6406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.401+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb7402e070 con 0x7fbb8406f780 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.401+0000 7fbb837fe640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 0x7fbb6406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.402+0000 7fbb837fe640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 0x7fbb6406dd00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb840b76d0 tx=0x7fbb6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.406+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb840b6ea0 con 0x7fbb8406f780 2024-03-24T10:34:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.585+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb840b6ea0 con 0x7fbb8406f780 2024-03-24T10:34:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.585+0000 7fbb80ff9640 1 -- 172.21.15.1:0/3778125459 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fbb840b6ea0 con 0x7fbb8406f780 2024-03-24T10:34:08.585 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 msgr2=0x7fbb6406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 0x7fbb6406dd00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb840b76d0 tx=0x7fbb6c009040 comp rx=0 tx=0).stop 2024-03-24T10:34:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 msgr2=0x7fbb840b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb740097d0 tx=0x7fbb740046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 shutdown_connections 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb6406b840 0x7fbb6406dd00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb8415e2c0 0x7fbb8415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb840cdd10 0x7fbb840c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 --2- 172.21.15.1:0/3778125459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb8406f780 0x7fbb840b5b00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 >> 172.21.15.1:0/3778125459 conn(0x7fbb840d1c60 msgr2=0x7fbb840af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 shutdown_connections 2024-03-24T10:34:08.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.587+0000 7fbb8a623640 1 -- 172.21.15.1:0/3778125459 wait complete. 2024-03-24T10:34:08.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.699+0000 7f7503a84640 1 Processor -- start 2024-03-24T10:34:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.699+0000 7f7503a84640 1 -- start start 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7503a84640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 0x7f74fc0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7503a84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 0x7f74fc06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7503a84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74fc0cdd60 con 0x7f74fc0b2aa0 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7503a84640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f74fc06eaa0 con 0x7f74fc144770 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7503a84640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f74fc071010 con 0x7f74fc070b90 2024-03-24T10:34:08.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7502021640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f74fc0b2aa0 0x7f74fc0ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41712/0 (socket says 172.21.15.1:41712) 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7502021640 1 -- 172.21.15.1:0/3419910169 learned_addr learned my addr 172.21.15.1:0/3419910169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750101f640 1 --2- 172.21.15.1:0/3419910169 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 0x7f74fc06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7501820640 1 --2- 172.21.15.1:0/3419910169 >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 0x7f74fc0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750101f640 1 -- 172.21.15.1:0/3419910169 >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 msgr2=0x7f74fc0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750101f640 1 --2- 172.21.15.1:0/3419910169 >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 0x7f74fc0d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750101f640 1 -- 172.21.15.1:0/3419910169 >> v1:172.21.15.1:6789/0 conn(0x7f74fc0b2aa0 legacy=0x7f74fc0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750081e640 1 -- 172.21.15.1:0/3419910169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688976906 0 0) 0x7f74fc0cdd60 con 0x7f74fc0b2aa0 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f750101f640 1 -- 172.21.15.1:0/3419910169 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74fc06df60 con 0x7f74fc070b90 2024-03-24T10:34:08.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.700+0000 7f7501820640 1 --2- 172.21.15.1:0/3419910169 >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 0x7f74fc0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f750101f640 1 --2- 172.21.15.1:0/3419910169 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 0x7f74fc06da20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0097a0 tx=0x7f74ec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7728a8ba405806d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f750081e640 1 -- 172.21.15.1:0/3419910169 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec031830 con 0x7f74fc070b90 2024-03-24T10:34:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f750081e640 1 -- 172.21.15.1:0/3419910169 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ec031990 con 0x7f74fc070b90 2024-03-24T10:34:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f750081e640 1 -- 172.21.15.1:0/3419910169 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec031d20 con 0x7f74fc070b90 2024-03-24T10:34:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f7503a84640 1 -- 172.21.15.1:0/3419910169 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 msgr2=0x7f74fc06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.701+0000 7f7503a84640 1 --2- 172.21.15.1:0/3419910169 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 0x7f74fc06da20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0097a0 tx=0x7f74ec02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 -- 172.21.15.1:0/3419910169 shutdown_connections 2024-03-24T10:34:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 --2- 172.21.15.1:0/3419910169 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc070b90 0x7f74fc06da20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 --2- 172.21.15.1:0/3419910169 >> v2:172.21.15.3:3300/0 conn(0x7f74fc144770 0x7f74fc0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 -- 172.21.15.1:0/3419910169 >> 172.21.15.1:0/3419910169 conn(0x7f74fc0b17a0 msgr2=0x7f74fc0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 -- 172.21.15.1:0/3419910169 shutdown_connections 2024-03-24T10:34:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.702+0000 7f7503a84640 1 -- 172.21.15.1:0/3419910169 wait complete. 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.703+0000 7f7503a84640 1 Processor -- start 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.703+0000 7f7503a84640 1 -- start start 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.703+0000 7f7503a84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.703+0000 7f7503a84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc0b2aa0 0x7f74fc0a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74fc144770 0x7f74fc13d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc13c5b0 con 0x7f74fc070b90 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc13d130 con 0x7f74fc144770 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f74fc15d7f0 con 0x7f74fc0b2aa0 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34010/0 (socket says 172.21.15.1:34010) 2024-03-24T10:34:08.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 -- 172.21.15.1:0/1174332159 learned_addr learned my addr 172.21.15.1:0/1174332159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7502021640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74fc144770 0x7f74fc13d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f750101f640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc0b2aa0 0x7f74fc0a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 -- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc0b2aa0 msgr2=0x7f74fc0a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc0b2aa0 0x7f74fc0a9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 -- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74fc144770 msgr2=0x7f74fc13d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74fc144770 0x7f74fc13d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 -- 172.21.15.1:0/1174332159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f8009400 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7501820640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e990 tx=0x7f74f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800c9e0 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f800cb40 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800ced0 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ec009400 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.704+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74fc13cae0 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.705+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74fc0b3aa0 con 0x7f74fc070b90 2024-03-24T10:34:08.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.706+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f74f8013470 con 0x7f74fc070b90 2024-03-24T10:34:08.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.706+0000 7f74f27fc640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 0x7f74e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:08.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.706+0000 7f750101f640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 0x7f74e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:08.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.706+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f74f808a860 con 0x7f74fc070b90 2024-03-24T10:34:08.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.707+0000 7f750101f640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 0x7f74e006dcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74ec036000 tx=0x7f74ec03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:08.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.710+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74f8055220 con 0x7f74fc070b90 2024-03-24T10:34:08.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.928+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74fc04f4e0 con 0x7f74fc070b90 2024-03-24T10:34:08.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.929+0000 7f74f27fc640 1 -- 172.21.15.1:0/1174332159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f74f80588e0 con 0x7f74fc070b90 2024-03-24T10:34:08.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:08.931 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:08.932 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 msgr2=0x7f74e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 0x7f74e006dcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74ec036000 tx=0x7f74ec03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 msgr2=0x7f74fc15d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e990 tx=0x7f74f800ee50 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 shutdown_connections 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f74e006b810 0x7f74e006dcd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74fc144770 0x7f74fc13d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f74fc0b2aa0 0x7f74fc0a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 --2- 172.21.15.1:0/1174332159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc070b90 0x7f74fc15d2b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 >> 172.21.15.1:0/1174332159 conn(0x7f74fc0b17a0 msgr2=0x7f74fc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 shutdown_connections 2024-03-24T10:34:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:08.931+0000 7f7503a84640 1 -- 172.21.15.1:0/1174332159 wait complete. 2024-03-24T10:34:08.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.044+0000 7f9c3b4fb640 1 Processor -- start 2024-03-24T10:34:09.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.044+0000 7f9c3b4fb640 1 -- start start 2024-03-24T10:34:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c3b4fb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9c3415bda0 0x7f9c3415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c3b4fb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 0x7f9c340cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c3b4fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3415d3c0 con 0x7f9c340cff00 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c3b4fb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9c340b2db0 con 0x7f9c3415bda0 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c3b4fb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3406e140 con 0x7f9c3406c910 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c39a98640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c340cff00 0x7f9c34071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41720/0 (socket says 172.21.15.1:41720) 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c39a98640 1 -- 172.21.15.1:0/3962592388 learned_addr learned my addr 172.21.15.1:0/3962592388 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c39297640 1 --2- 172.21.15.1:0/3962592388 >> v2:172.21.15.3:3300/0 conn(0x7f9c3415bda0 0x7f9c3415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.045+0000 7f9c38a96640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 0x7f9c340cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c2bfff640 1 -- 172.21.15.1:0/3962592388 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057257317 0 0) 0x7f9c3415d3c0 con 0x7f9c340cff00 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c2bfff640 1 -- 172.21.15.1:0/3962592388 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c20003520 con 0x7f9c340cff00 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c39297640 1 -- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 msgr2=0x7f9c340cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c39297640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 0x7f9c340cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c39297640 1 -- 172.21.15.1:0/3962592388 >> v1:172.21.15.1:6789/0 conn(0x7f9c340cff00 legacy=0x7f9c34071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c39297640 1 -- 172.21.15.1:0/3962592388 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3406ce40 con 0x7f9c3415bda0 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.046+0000 7f9c38a96640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 0x7f9c340cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c39297640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c3415bda0 0x7f9c3415ce80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c300097a0 tx=0x7f9c3002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e666756707cec67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c2bfff640 1 -- 172.21.15.1:0/3962592388 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c30031780 con 0x7f9c3415bda0 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c2bfff640 1 -- 172.21.15.1:0/3962592388 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c300318e0 con 0x7f9c3415bda0 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c2bfff640 1 -- 172.21.15.1:0/3962592388 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c30031c50 con 0x7f9c3415bda0 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c3415bda0 msgr2=0x7f9c3415ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c3415bda0 0x7f9c3415ce80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c300097a0 tx=0x7f9c3002b820 comp rx=0 tx=0).stop 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3962592388 shutdown_connections 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3406c910 0x7f9c340cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3962592388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c3415bda0 0x7f9c3415ce80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3962592388 >> 172.21.15.1:0/3962592388 conn(0x7f9c340b17a0 msgr2=0x7f9c340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3962592388 shutdown_connections 2024-03-24T10:34:09.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.047+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3962592388 wait complete. 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.048+0000 7f9c3b4fb640 1 Processor -- start 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.048+0000 7f9c3b4fb640 1 -- start start 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c340cff00 0x7f9c340cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3415bda0 0x7f9c340cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9c340a9430 con 0x7f9c3406c910 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9c340a9fb0 con 0x7f9c340cff00 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c3b4fb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9c34155c10 con 0x7f9c3415bda0 2024-03-24T10:34:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39297640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34020/0 (socket says 172.21.15.1:34020) 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39297640 1 -- 172.21.15.1:0/3711380641 learned_addr learned my addr 172.21.15.1:0/3711380641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c38a96640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c340cff00 0x7f9c340cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39a98640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3415bda0 0x7f9c340cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39297640 1 -- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3415bda0 msgr2=0x7f9c340cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.049+0000 7f9c39297640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3415bda0 0x7f9c340cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c39297640 1 -- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c340cff00 msgr2=0x7f9c340cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c39297640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c340cff00 0x7f9c340cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c39297640 1 -- 172.21.15.1:0/3711380641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c24009400 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c39297640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098c0 tx=0x7f9c300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c30032820 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c30032980 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c30009400 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c30032cf0 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.050+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c340a9960 con 0x7f9c3406c910 2024-03-24T10:34:09.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.051+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c34155d50 con 0x7f9c3406c910 2024-03-24T10:34:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.051+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9c3004e440 con 0x7f9c3406c910 2024-03-24T10:34:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.052+0000 7f9c29ffb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 0x7f9c1806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.052+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c300ac000 con 0x7f9c3406c910 2024-03-24T10:34:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.052+0000 7f9c38a96640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 0x7f9c1806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.052+0000 7f9c38a96640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 0x7f9c1806dd00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c3415aa00 tx=0x7f9c24009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.056+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9c34155d50 con 0x7f9c3406c910 2024-03-24T10:34:09.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.239+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c34155d50 con 0x7f9c3406c910 2024-03-24T10:34:09.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.240+0000 7f9c29ffb640 1 -- 172.21.15.1:0/3711380641 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f9c34155d50 con 0x7f9c3406c910 2024-03-24T10:34:09.240 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:09.240 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 msgr2=0x7f9c1806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 0x7f9c1806dd00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c3415aa00 tx=0x7f9c24009190 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 msgr2=0x7f9c341556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098c0 tx=0x7f9c300046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 shutdown_connections 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9c1806b840 0x7f9c1806dd00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9c3415bda0 0x7f9c340cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c340cff00 0x7f9c340cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 --2- 172.21.15.1:0/3711380641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c3406c910 0x7f9c341556d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 >> 172.21.15.1:0/3711380641 conn(0x7f9c340b17a0 msgr2=0x7f9c340ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 shutdown_connections 2024-03-24T10:34:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.242+0000 7f9c3b4fb640 1 -- 172.21.15.1:0/3711380641 wait complete. 2024-03-24T10:34:09.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.354+0000 7f054395c640 1 Processor -- start 2024-03-24T10:34:09.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.355+0000 7f054395c640 1 -- start start 2024-03-24T10:34:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.355+0000 7f054395c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f053c0cebc0 0x7f053c0d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.355+0000 7f054395c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c0d4eb0 con 0x7f053c15e0a0 2024-03-24T10:34:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.355+0000 7f054395c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c0cf0e0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.355+0000 7f054395c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f053c0d0160 con 0x7f053c0cebc0 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f05416f8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f053c15e0a0 0x7f053c0d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41730/0 (socket says 172.21.15.1:41730) 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f05416f8640 1 -- 172.21.15.1:0/198542375 learned_addr learned my addr 172.21.15.1:0/198542375 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0541ef9640 1 --2- 172.21.15.1:0/198542375 >> v2:172.21.15.3:3300/0 conn(0x7f053c0cebc0 0x7f053c0d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526547190 0 0) 0x7f0538001110 con 0x7f053c15e0a0 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0528003520 con 0x7f053c15e0a0 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0541ef9640 1 -- 172.21.15.1:0/198542375 >> v1:172.21.15.3:6789/0 conn(0x7f053c0d2ca0 legacy=0x7f053c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410995689 0 0) 0x7f053c0cf0e0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0541ef9640 1 -- 172.21.15.1:0/198542375 >> v1:172.21.15.1:6789/0 conn(0x7f053c15e0a0 legacy=0x7f053c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.356+0000 7f0541ef9640 1 -- 172.21.15.1:0/198542375 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c0cf3c0 con 0x7f053c0cebc0 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f0541ef9640 1 --2- 172.21.15.1:0/198542375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0d4970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0534009d90 tx=0x7f053402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa0da33c227b9ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053402cd80 con 0x7f053c0cebc0 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0534034440 con 0x7f053c0cebc0 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f0533fff640 1 -- 172.21.15.1:0/198542375 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05340347d0 con 0x7f053c0cebc0 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f054395c640 1 -- 172.21.15.1:0/198542375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 msgr2=0x7f053c0d4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f054395c640 1 --2- 172.21.15.1:0/198542375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0d4970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0534009d90 tx=0x7f053402bfd0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f054395c640 1 -- 172.21.15.1:0/198542375 shutdown_connections 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f054395c640 1 --2- 172.21.15.1:0/198542375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0d4970 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.357+0000 7f054395c640 1 -- 172.21.15.1:0/198542375 >> 172.21.15.1:0/198542375 conn(0x7f053c09d1c0 msgr2=0x7f053c0d3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.358+0000 7f054395c640 1 -- 172.21.15.1:0/198542375 shutdown_connections 2024-03-24T10:34:09.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.358+0000 7f054395c640 1 -- 172.21.15.1:0/198542375 wait complete. 2024-03-24T10:34:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 Processor -- start 2024-03-24T10:34:09.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 -- start start 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f053c15e0a0 0x7f053c14d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f053c0b5710 con 0x7f053c0d2ca0 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f053c0b4670 con 0x7f053c0cebc0 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f054395c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f053c0b4950 con 0x7f053c15e0a0 2024-03-24T10:34:09.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f0540ef7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f0540ef7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34028/0 (socket says 172.21.15.1:34028) 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f0540ef7640 1 -- 172.21.15.1:0/1401807747 learned_addr learned my addr 172.21.15.1:0/1401807747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.359+0000 7f05416f8640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0541ef9640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f053c15e0a0 0x7f053c14d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 -- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f053c15e0a0 msgr2=0x7f053c14d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f053c15e0a0 0x7f053c14d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 -- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 msgr2=0x7f053c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 -- 172.21.15.1:0/1401807747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0534009a70 con 0x7f053c0d2ca0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0540ef7640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c4b0 tx=0x7f052c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c005480 con 0x7f053c0d2ca0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f052c0035f0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c0b4c30 con 0x7f053c0d2ca0 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c0039e0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.360+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c0b51f0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.361+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c069350 con 0x7f053c0d2ca0 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.362+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f052c003b40 con 0x7f053c0d2ca0 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.362+0000 7f0531ffb640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:09.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.362+0000 7f05416f8640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:09.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.362+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f052c0897f0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.362+0000 7f05416f8640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 0x7f052006dcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0538005880 tx=0x7f0538009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:09.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.366+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f052c054220 con 0x7f053c0d2ca0 2024-03-24T10:34:09.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.587+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f053c04f4e0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.588+0000 7f0531ffb640 1 -- 172.21.15.1:0/1401807747 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f052c0578e0 con 0x7f053c0d2ca0 2024-03-24T10:34:09.588 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:09.588 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:09.588 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 msgr2=0x7f052006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 0x7f052006dcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0538005880 tx=0x7f0538009040 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 msgr2=0x7f053c0cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c4b0 tx=0x7f052c00c970 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 shutdown_connections 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f053c15e0a0 0x7f053c14d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f053c0d2ca0 0x7f053c0cb260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 --2- 172.21.15.1:0/1401807747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f053c0cebc0 0x7f053c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 >> 172.21.15.1:0/1401807747 conn(0x7f053c09d1c0 msgr2=0x7f053c06d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 shutdown_connections 2024-03-24T10:34:09.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:09.590+0000 7f054395c640 1 -- 172.21.15.1:0/1401807747 wait complete. 2024-03-24T10:34:10.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:33:54.483199+0000', 'modified': '2024-03-24T10:33:57.542563+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6834/824939987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6832', 'nonce': 824939987}, {'type': 'v1', 'addr': '172.21.15.3:6834', 'nonce': 824939987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4287, 'qdb_cluster': [4287]} 2024-03-24T10:34:10.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T10:34:10.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T10:34:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.111+0000 7fcc7da39640 1 Processor -- start 2024-03-24T10:34:10.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.111+0000 7fcc7da39640 1 -- start start 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 0x7fcc7815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc7806c910 0x7fcc780cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcc780cff00 0x7fcc78071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcc780b2db0 con 0x7fcc7806c910 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fcc7806e140 con 0x7fcc780cff00 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc7da39640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcc780d0390 con 0x7fcc7815bda0 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc7806c910 0x7fcc780cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 0x7fcc7815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc7806c910 0x7fcc780cf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34038/0 (socket says 172.21.15.1:34038) 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 -- 172.21.15.1:0/4134592138 learned_addr learned my addr 172.21.15.1:0/4134592138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc777fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcc780cff00 0x7fcc78071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 -- 172.21.15.1:0/4134592138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 msgr2=0x7fcc7815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 0x7fcc7815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 -- 172.21.15.1:0/4134592138 >> v2:172.21.15.3:3300/0 conn(0x7fcc780cff00 msgr2=0x7fcc78071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/4134592138 >> v2:172.21.15.3:3300/0 conn(0x7fcc780cff00 0x7fcc78071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc6ffff640 1 -- 172.21.15.1:0/4134592138 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7815d3c0 con 0x7fcc7806c910 2024-03-24T10:34:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.112+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 0x7fcc7815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.113+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7806c910 0x7fcc780cf9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc780cc450 tx=0x7fcc6002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d4b41a1ba7b463 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.113+0000 7fcc767fc640 1 -- 172.21.15.1:0/4134592138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc600317c0 con 0x7fcc7806c910 2024-03-24T10:34:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.113+0000 7fcc767fc640 1 -- 172.21.15.1:0/4134592138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60031920 con 0x7fcc7806c910 2024-03-24T10:34:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.113+0000 7fcc767fc640 1 -- 172.21.15.1:0/4134592138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60031cb0 con 0x7fcc7806c910 2024-03-24T10:34:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 -- 172.21.15.1:0/4134592138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7806c910 msgr2=0x7fcc780cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7806c910 0x7fcc780cf9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc780cc450 tx=0x7fcc6002b860 comp rx=0 tx=0).stop 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 -- 172.21.15.1:0/4134592138 shutdown_connections 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 --2- 172.21.15.1:0/4134592138 >> v2:172.21.15.3:3300/0 conn(0x7fcc780cff00 0x7fcc78071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7806c910 0x7fcc780cf9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 --2- 172.21.15.1:0/4134592138 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7815bda0 0x7fcc7815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 -- 172.21.15.1:0/4134592138 >> 172.21.15.1:0/4134592138 conn(0x7fcc780b17a0 msgr2=0x7fcc780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 -- 172.21.15.1:0/4134592138 shutdown_connections 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.114+0000 7fcc7da39640 1 -- 172.21.15.1:0/4134592138 wait complete. 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.115+0000 7fcc7da39640 1 Processor -- start 2024-03-24T10:34:10.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.115+0000 7fcc7da39640 1 -- start start 2024-03-24T10:34:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 0x7fcc78151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc780cff00 0x7fcc781e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcc781e6980 con 0x7fcc7815bda0 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fcc781e6f00 con 0x7fcc780cff00 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc7da39640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcc78151680 con 0x7fcc7806c910 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34050/0 (socket says 172.21.15.1:34050) 2024-03-24T10:34:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 -- 172.21.15.1:0/1194258604 learned_addr learned my addr 172.21.15.1:0/1194258604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 0x7fcc78151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc780cff00 0x7fcc781e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 -- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 msgr2=0x7fcc78151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 0x7fcc78151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 -- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc780cff00 msgr2=0x7fcc781e4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc780cff00 0x7fcc781e4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 -- 172.21.15.1:0/1194258604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc68009400 con 0x7fcc7815bda0 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 0x7fcc78151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:10.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.116+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000efc0 tx=0x7fcc7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.117+0000 7fcc74ff9640 1 -- 172.21.15.1:0/1194258604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70014480 con 0x7fcc7815bda0 2024-03-24T10:34:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.117+0000 7fcc74ff9640 1 -- 172.21.15.1:0/1194258604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc7000f040 con 0x7fcc7815bda0 2024-03-24T10:34:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.117+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc600094d0 con 0x7fcc7815bda0 2024-03-24T10:34:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.117+0000 7fcc74ff9640 1 -- 172.21.15.1:0/1194258604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70014820 con 0x7fcc7815bda0 2024-03-24T10:34:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.117+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc781f5dc0 con 0x7fcc7815bda0 2024-03-24T10:34:10.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.119+0000 7fcc74ff9640 1 -- 172.21.15.1:0/1194258604 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc70014980 con 0x7fcc7815bda0 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.119+0000 7fcc74ff9640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 0x7fcc5806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.119+0000 7fcc74ff9640 1 -- 172.21.15.1:0/1194258604 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc7008ac20 con 0x7fcc7815bda0 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.119+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 0x7fcc5806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.119+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 0x7fcc5806dcd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009fa0 tx=0x7fcc68009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.120+0000 7fcc80618740 1 Processor -- start 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.120+0000 7fcc80618740 1 -- start start 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f51af0 0x5565b8f818e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565b8ec5180 con 0x5565b8f7e9f0 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5565b8ec26b0 con 0x5565b8f7e610 2024-03-24T10:34:10.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5565b8ebde90 con 0x5565b8f51af0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:50206/0 (socket says 172.21.15.1:50206) 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc76ffd640 1 -- 172.21.15.1:0/155982282 learned_addr learned my addr 172.21.15.1:0/155982282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc777fe640 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f51af0 0x5565b8f818e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 -- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f51af0 msgr2=0x5565b8f818e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f51af0 0x5565b8f818e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 -- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 msgr2=0x5565b8f515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 -- 172.21.15.1:0/155982282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565b8e7ad80 con 0x5565b8f7e9f0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f813a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc78073490 tx=0x7fcc60008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9702072d3044c218 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6d7fa640 1 -- 172.21.15.1:0/155982282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60031a90 con 0x5565b8f7e9f0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6d7fa640 1 -- 172.21.15.1:0/155982282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60031bf0 con 0x5565b8f7e9f0 2024-03-24T10:34:10.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.121+0000 7fcc6d7fa640 1 -- 172.21.15.1:0/155982282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60046480 con 0x5565b8f7e9f0 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 -- 172.21.15.1:0/155982282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 msgr2=0x5565b8f813a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f813a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc78073490 tx=0x7fcc60008030 comp rx=0 tx=0).stop 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 -- 172.21.15.1:0/155982282 shutdown_connections 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f51af0 0x5565b8f818e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f813a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 --2- 172.21.15.1:0/155982282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 -- 172.21.15.1:0/155982282 >> 172.21.15.1:0/155982282 conn(0x5565b8f53820 msgr2=0x5565b8f9c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 -- 172.21.15.1:0/155982282 shutdown_connections 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.122+0000 7fcc80618740 1 -- 172.21.15.1:0/155982282 wait complete. 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.123+0000 7fcc80618740 1 -- start start 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.123+0000 7fcc80618740 1 Processor -- start 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e610 0x5565b8f84a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f83430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565b8e62d50 con 0x5565b8f7e9f0 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5565b8b91310 con 0x5565b8f6e270 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5565b8e438f0 con 0x5565b8f7e610 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc80618740 10 client.0 _ll_register_callbacks cb 0x5565b8bceac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e610 0x5565b8f84a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34058/0 (socket says 172.21.15.1:34058) 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1341011939 learned_addr learned my addr 172.21.15.1:0/1341011939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f83430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e610 msgr2=0x5565b8f84a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e610 0x5565b8f84a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 msgr2=0x5565b8f83430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f83430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1341011939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7815d3c0 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.124+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009990 tx=0x7fcc60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bebe00bc5d64c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60004230 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565b8bbd6c0 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc60040070 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcc60004390 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 10 client.4710 fetch_fsmap learned FSMap version 8 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5565b8ebc510 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5565b8bbd6c0 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7fcc60057070 con 0x5565b8f7e9f0 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 10 client.4710 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 10 client.4710 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:34:10.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 20 client.4710 populate_metadata read hostname 'smithi001' 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8fb9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 4 client.4710 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 --> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565b8c0e270 con 0x5565b8f83c70 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.125+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8fb9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.126+0000 7fcc567fc640 10 client.4710 ms_handle_connect on v2:172.21.15.3:6832/824939987 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.126+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8fb9470 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f3f46fd899ff822 server_cookie=cc354c1bcd5d7467 in_seq=0 out_seq=0 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc567fc640 1 -- 172.21.15.1:0/1341011939 <== mds.0 v2:172.21.15.3:6832/824939987 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565b8c0e270 con 0x5565b8f83c70 2024-03-24T10:34:10.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc567fc640 10 client.4710 handle_command_reply: tid=0 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 client.4710 shutdown 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 msgr2=0x5565b8f84fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009990 tx=0x7fcc60004730 comp rx=0 tx=0).stop 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 shutdown_connections 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8fb9470 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f83430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc567fc640 0 client.4710 ms_handle_reset on v2:172.21.15.3:6832/824939987 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e9f0 0x5565b8f84fb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 --2- 172.21.15.1:0/1341011939 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e610 0x5565b8f84a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 >> 172.21.15.1:0/1341011939 conn(0x5565b8f51af0 msgr2=0x5565b8f574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 shutdown_connections 2024-03-24T10:34:10.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 1 -- 172.21.15.1:0/1341011939 wait complete. 2024-03-24T10:34:10.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.127+0000 7fcc80618740 20 client.4710 trim_cache size 0 max 16384 2024-03-24T10:34:10.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.145+0000 7fcc80618740 1 Processor -- start 2024-03-24T10:34:10.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.145+0000 7fcc80618740 1 -- start start 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f9f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f6acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565b8fbe630 con 0x5565b8f7e610 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5565b8f50180 con 0x5565b8f6e270 2024-03-24T10:34:10.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5565b8f58a90 con 0x5565b8f7e9f0 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34074/0 (socket says 172.21.15.1:34074) 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1507209920 learned_addr learned my addr 172.21.15.1:0/1507209920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f9f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f6acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 msgr2=0x5565b8f6acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f6acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 msgr2=0x5565b8f9f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f9f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc6ffff640 1 -- 172.21.15.1:0/1507209920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565b8e58a80 con 0x5565b8f7e610 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.146+0000 7fcc777fe640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f6acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcc600097f0 tx=0x7fcc6004d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49877c1c95adc7c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc6cff9640 1 -- 172.21.15.1:0/1507209920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc600069c0 con 0x5565b8f7e610 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc6cff9640 1 -- 172.21.15.1:0/1507209920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60003a70 con 0x5565b8f7e610 2024-03-24T10:34:10.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc6cff9640 1 -- 172.21.15.1:0/1507209920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60003de0 con 0x5565b8f7e610 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 -- 172.21.15.1:0/1507209920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 msgr2=0x5565b8f9fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcc600097f0 tx=0x7fcc6004d000 comp rx=0 tx=0).stop 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 -- 172.21.15.1:0/1507209920 shutdown_connections 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f6acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f7e610 0x5565b8f9fdf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 --2- 172.21.15.1:0/1507209920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f6e270 0x5565b8f9f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 -- 172.21.15.1:0/1507209920 >> 172.21.15.1:0/1507209920 conn(0x5565b8f51af0 msgr2=0x5565b8f4c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 -- 172.21.15.1:0/1507209920 shutdown_connections 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.147+0000 7fcc80618740 1 -- 172.21.15.1:0/1507209920 wait complete. 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.148+0000 7fcc80618740 1 -- start start 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.148+0000 7fcc80618740 1 Processor -- start 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f6b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f9f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565b8f98810 con 0x5565b8f6e270 2024-03-24T10:34:10.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5565b8f6b2f0 con 0x5565b8f7e610 2024-03-24T10:34:10.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5565b8f50bc0 con 0x5565b8f7e9f0 2024-03-24T10:34:10.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc80618740 10 client.0 _ll_register_callbacks cb 0x7fcc60049070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34078/0 (socket says 172.21.15.1:34078) 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 -- 172.21.15.1:0/457590338 learned_addr learned my addr 172.21.15.1:0/457590338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f6b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc777fe640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f9f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 -- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 msgr2=0x5565b8f9f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f9f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 -- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 msgr2=0x5565b8f6b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f6b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 -- 172.21.15.1:0/457590338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565b8ebc510 con 0x5565b8f6e270 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc76ffd640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcc68004650 tx=0x7fcc68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646404aab34e4801 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc567fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.149+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68029040 con 0x5565b8f6e270 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc68003e30 con 0x5565b8f6e270 2024-03-24T10:34:10.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc60031bf0 con 0x5565b8f6e270 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcc6801c040 con 0x5565b8f6e270 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 10 client.4716 fetch_fsmap learned FSMap version 8 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5565b8e7e7e0 con 0x5565b8f6e270 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60031bf0 con 0x5565b8f6e270 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 8) ==== 3494+0+0 (secure 0 0 0) 0x7fcc6802db10 con 0x5565b8f6e270 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 10 client.4716 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 10 client.4716 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:34:10.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 20 client.4716 populate_metadata read hostname 'smithi001' 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8f9fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 4 client.4716 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 --> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] -- command(tid 0: {"prefix": "status"}) -- 0x5565b8bfa610 con 0x5565b8f83c70 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8f9fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc6ffff640 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8f9fdf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ffd52460055555a server_cookie=5ddbde97a47786d0 in_seq=0 out_seq=0 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.150+0000 7fcc567fc640 10 client.4716 ms_handle_connect on v2:172.21.15.3:6832/824939987 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc567fc640 1 -- 172.21.15.1:0/457590338 <== mds.0 v2:172.21.15.3:6832/824939987 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5565b8bfa610 con 0x5565b8f83c70 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc567fc640 10 client.4716 handle_command_reply: tid=0 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 client.4716 shutdown 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 msgr2=0x5565b8f6acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcc68004650 tx=0x7fcc68004730 comp rx=0 tx=0).stop 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 shutdown_connections 2024-03-24T10:34:10.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:6832/824939987,v1:172.21.15.3:6834/824939987] conn(0x5565b8f83c70 0x5565b8f9fdf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5565b8f7e9f0 0x5565b8f9f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565b8f7e610 0x5565b8f6b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc567fc640 0 client.4716 ms_handle_reset on v2:172.21.15.3:6832/824939987 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 --2- 172.21.15.1:0/457590338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565b8f6e270 0x5565b8f6acf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 >> 172.21.15.1:0/457590338 conn(0x5565b8f51af0 msgr2=0x5565b8f85200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 shutdown_connections 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 1 -- 172.21.15.1:0/457590338 wait complete. 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc80618740 20 client.4716 trim_cache size 0 max 16384 2024-03-24T10:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "a0192ab9-51d5-4157-8f81-4a83111a7dfa", 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4287, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 15.019593531, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 8, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-03-24T10:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 22.063426935999999 2024-03-24T10:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 msgr2=0x7fcc5806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 0x7fcc5806dcd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009fa0 tx=0x7fcc68009190 comp rx=0 tx=0).stop 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 msgr2=0x7fcc781f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000efc0 tx=0x7fcc7000c490 comp rx=0 tx=0).stop 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 shutdown_connections 2024-03-24T10:34:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcc5806b810 0x7fcc5806dcd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc7815bda0 0x7fcc781f5020 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc780cff00 0x7fcc781e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 --2- 172.21.15.1:0/1194258604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc7806c910 0x7fcc78151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 >> 172.21.15.1:0/1194258604 conn(0x7fcc780b17a0 msgr2=0x7fcc7815ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 shutdown_connections 2024-03-24T10:34:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.151+0000 7fcc7da39640 1 -- 172.21.15.1:0/1194258604 wait complete. 2024-03-24T10:34:10.164 INFO:teuthology.run_tasks:Running task sequential... 2024-03-24T10:34:10.176 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-24T10:34:10.176 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-24T10:34:10.177 INFO:teuthology.task.print:Enabling mgr modules 2024-03-24T10:34:10.177 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-24T10:34:10.206 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-24T10:34:10.206 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-24T10:34:10.207 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:10.207 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:10.207 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:10.207 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:34:10.207 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:10.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.208 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:34:10.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-24T10:34:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:Cannot find device "ceph-brx" 2024-03-24T10:34:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:34:10.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.293 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-03-24T10:34:10.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-03-24T10:34:10.376 INFO:teuthology.orchestra.run.smithi003.stderr:Cannot find device "ceph-brx" 2024-03-24T10:34:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:34:10.378 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-24T10:34:10.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:34:10.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.480+0000 7f674599a640 1 Processor -- start 2024-03-24T10:34:10.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.480+0000 7f674599a640 1 -- start start 2024-03-24T10:34:10.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f674599a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67400b31b0 0x7f6740070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f674599a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67400cdd60 con 0x7f6740144770 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f674599a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674006e200 con 0x7f67400d00f0 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f674599a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f674006eaa0 con 0x7f67400b31b0 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6740144770 0x7f674006da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41736/0 (socket says 172.21.15.1:41736) 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673effd640 1 -- 172.21.15.1:0/2383181567 learned_addr learned my addr 172.21.15.1:0/2383181567 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673f7fe640 1 --2- 172.21.15.1:0/2383181567 >> v2:172.21.15.3:3300/0 conn(0x7f67400b31b0 0x7f6740070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344457327 0 0) 0x7f67400cdd60 con 0x7f6740144770 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f672c003520 con 0x7f6740144770 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855375420 0 0) 0x7f674006e200 con 0x7f67400d00f0 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67400cdd60 con 0x7f67400d00f0 2024-03-24T10:34:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673f7fe640 1 -- 172.21.15.1:0/2383181567 >> v1:172.21.15.3:6789/0 conn(0x7f67400d00f0 legacy=0x7f67400b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673f7fe640 1 -- 172.21.15.1:0/2383181567 >> v1:172.21.15.1:6789/0 conn(0x7f6740144770 legacy=0x7f674006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.481+0000 7f673f7fe640 1 -- 172.21.15.1:0/2383181567 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674006e4e0 con 0x7f67400b31b0 2024-03-24T10:34:10.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.482+0000 7f673f7fe640 1 --2- 172.21.15.1:0/2383181567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400b31b0 0x7f6740070b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6738009d90 tx=0x7f673802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=398e9af41c84a684 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.482+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673802cd00 con 0x7f67400b31b0 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.482+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f673802ce60 con 0x7f67400b31b0 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.482+0000 7f673e7fc640 1 -- 172.21.15.1:0/2383181567 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738034640 con 0x7f67400b31b0 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 -- 172.21.15.1:0/2383181567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400b31b0 msgr2=0x7f6740070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 --2- 172.21.15.1:0/2383181567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400b31b0 0x7f6740070b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6738009d90 tx=0x7f673802bf50 comp rx=0 tx=0).stop 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 -- 172.21.15.1:0/2383181567 shutdown_connections 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 --2- 172.21.15.1:0/2383181567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400b31b0 0x7f6740070b90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 -- 172.21.15.1:0/2383181567 >> 172.21.15.1:0/2383181567 conn(0x7f67400b17a0 msgr2=0x7f67400b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 -- 172.21.15.1:0/2383181567 shutdown_connections 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.483+0000 7f674599a640 1 -- 172.21.15.1:0/2383181567 wait complete. 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.484+0000 7f674599a640 1 Processor -- start 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.484+0000 7f674599a640 1 -- start start 2024-03-24T10:34:10.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400d00f0 0x7f6740151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6740144770 0x7f67401f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f674013e980 con 0x7f67400b31b0 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6740159d30 con 0x7f67400d00f0 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f674599a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f674013dd40 con 0x7f6740144770 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34092/0 (socket says 172.21.15.1:34092) 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 -- 172.21.15.1:0/534008822 learned_addr learned my addr 172.21.15.1:0/534008822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f67367fc640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400d00f0 0x7f6740151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673f7fe640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6740144770 0x7f67401f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 -- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6740144770 msgr2=0x7f67401f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6740144770 0x7f67401f1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 -- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400d00f0 msgr2=0x7f6740151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400d00f0 0x7f6740151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 -- 172.21.15.1:0/534008822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738009a70 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673effd640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f673000c410 tx=0x7f673000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730005ce0 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6730005e40 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.485+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300072d0 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.486+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674013dfe0 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.486+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674013e5a0 con 0x7f67400b31b0 2024-03-24T10:34:10.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.487+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6740069550 con 0x7f67400b31b0 2024-03-24T10:34:10.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.487+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6730007430 con 0x7f67400b31b0 2024-03-24T10:34:10.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.487+0000 7f673cff9640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 0x7f672006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:10.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.487+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6730089620 con 0x7f67400b31b0 2024-03-24T10:34:10.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.487+0000 7f67367fc640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 0x7f672006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:10.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.488+0000 7f67367fc640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 0x7f672006dd00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:10.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.492+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6740069550 con 0x7f67400b31b0 2024-03-24T10:34:10.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.713+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6740069550 con 0x7f67400b31b0 2024-03-24T10:34:10.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.713+0000 7f673cff9640 1 -- 172.21.15.1:0/534008822 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f6740069550 con 0x7f67400b31b0 2024-03-24T10:34:10.714 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:34:10.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 msgr2=0x7f672006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 0x7f672006dd00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728009040 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 msgr2=0x7f674015dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f673000c410 tx=0x7f673000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 shutdown_connections 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f672006b840 0x7f672006dd00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6740144770 0x7f67401f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67400d00f0 0x7f6740151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 --2- 172.21.15.1:0/534008822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67400b31b0 0x7f674015dc30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 >> 172.21.15.1:0/534008822 conn(0x7f67400b17a0 msgr2=0x7f6740142d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 shutdown_connections 2024-03-24T10:34:10.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10.715+0000 7f674599a640 1 -- 172.21.15.1:0/534008822 wait complete. 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:34:10.729 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:34:10.730 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:34:10.730 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:34:10.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.730 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:10.745 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42690sec preferred_lft 42690sec 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:10.746 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:34:10.747 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T10:34:10.747 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:10.747 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:10.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-03-24T10:34:10.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-03-24T10:34:10.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-03-24T10:34:10.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T10:34:10.748 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:10.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:10.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.950 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:34:10.976 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:34:10.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:10.979 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:34:11.050 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:34:11.050 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:34:11.051 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:34:11.051 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:34:11.051 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:34:11.051 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:11.051 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:11.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:34:11.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:34:11.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:34:11.052 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:11.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:11.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:11.340 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:34:11.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:11.356 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-24T10:34:11.411 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:11.411 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:11.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:11.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T10:34:11.411 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:11.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:11.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:11.536 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T10:34:11.536 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:11.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:11.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:11.675 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:11.675 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:11.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:34:11.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T10:34:11.675 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:11.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:11.763 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:34:11.763 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:11.763 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:11.779 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:34:11.779 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:11.779 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:11.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:34:11.904 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:11.959 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:34:11.959 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:34:11.959 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:34:11.959 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:34:11.960 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:11.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:11.962 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:34:12.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:34:12.015 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:34:12.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:34:12.084 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:34:12.140 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:34:12.141 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:12.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:12.142 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:34:12.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:34:12.144+0000 7f4473243480 -1 init, newargv = 0x55b6899704e0 newargc=15 2024-03-24T10:34:12.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[46484]: starting ceph client 2024-03-24T10:34:12.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[46484]: starting fuse 2024-03-24T10:34:12.163 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:34:12.164 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:12.164 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:34:12.165 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:34:12.166 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:12.166 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:34:12.166 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:34:12.285 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.46484.asok 2024-03-24T10:34:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:12.296 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46484.asok 2024-03-24T10:34:12.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:12.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46484.asok status 2024-03-24T10:34:12.433 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:34:12.433 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:34:12.433 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "46484", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4728, 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4728 2024-03-24T10:34:12.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 136262510 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 136262510 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4728 192.168.0.1:0/136262510", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/136262510", 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 8, 2024-03-24T10:34:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 21, 2024-03-24T10:34:12.436 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:34:12.436 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:34:12.436 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:34:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:34:12.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:34:12.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.542+0000 7f3ed1bc2640 1 Processor -- start 2024-03-24T10:34:12.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.542+0000 7f3ed1bc2640 1 -- start start 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ed1bc2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc0cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ed1bc2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ecc0cff00 0x7f3ecc165a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ed1bc2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc06ce40 con 0x7f3ecc15bda0 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ed1bc2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3ecc0b2db0 con 0x7f3ecc0cff00 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ed1bc2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ecc06e140 con 0x7f3ecc06c910 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecb7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ecc15bda0 0x7f3ecc15ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59628/0 (socket says 172.21.15.1:59628) 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecb7fe640 1 -- 172.21.15.1:0/4194946018 learned_addr learned my addr 172.21.15.1:0/4194946018 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/4194946018 >> v2:172.21.15.1:3300/0 conn(0x7f3ecc0cff00 0x7f3ecc165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecaffd640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc0cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3eca7fc640 1 -- 172.21.15.1:0/4194946018 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651864143 0 0) 0x7f3ecc06ce40 con 0x7f3ecc15bda0 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3eca7fc640 1 -- 172.21.15.1:0/4194946018 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eb4003520 con 0x7f3ecc15bda0 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecbfff640 1 -- 172.21.15.1:0/4194946018 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 msgr2=0x7f3ecc0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc0cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecbfff640 1 -- 172.21.15.1:0/4194946018 >> v1:172.21.15.1:6789/0 conn(0x7f3ecc15bda0 legacy=0x7f3ecc15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.543+0000 7f3ecbfff640 1 -- 172.21.15.1:0/4194946018 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc06d120 con 0x7f3ecc0cff00 2024-03-24T10:34:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.544+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc165a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009d90 tx=0x7f3ec402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67176626c60e83c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:12.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.544+0000 7f3eca7fc640 1 -- 172.21.15.1:0/4194946018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec402cbc0 con 0x7f3ecc0cff00 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.544+0000 7f3eca7fc640 1 -- 172.21.15.1:0/4194946018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ec402cd20 con 0x7f3ecc0cff00 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.544+0000 7f3eca7fc640 1 -- 172.21.15.1:0/4194946018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4034660 con 0x7f3ecc0cff00 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/4194946018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc0cff00 msgr2=0x7f3ecc165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc165a20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009d90 tx=0x7f3ec402be10 comp rx=0 tx=0).stop 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/4194946018 shutdown_connections 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc0cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/4194946018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc165a20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/4194946018 >> 172.21.15.1:0/4194946018 conn(0x7f3ecc0b17a0 msgr2=0x7f3ecc0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/4194946018 shutdown_connections 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.545+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/4194946018 wait complete. 2024-03-24T10:34:12.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.546+0000 7f3ed1bc2640 1 Processor -- start 2024-03-24T10:34:12.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.546+0000 7f3ed1bc2640 1 -- start start 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc0a97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1e2650 con 0x7f3ecc15bda0 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1e2930 con 0x7f3ecc0cff00 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ed1bc2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ecc1e2c10 con 0x7f3ecc06c910 2024-03-24T10:34:12.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58642/0 (socket says 172.21.15.1:58642) 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 -- 172.21.15.1:0/2646611257 learned_addr learned my addr 172.21.15.1:0/2646611257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecb7fe640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecaffd640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc0a97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 -- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 msgr2=0x7f3ecc153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 -- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ecc0cff00 msgr2=0x7f3ecc0a97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc0a97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.547+0000 7f3ecbfff640 1 -- 172.21.15.1:0/2646611257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec4009a70 con 0x7f3ecc15bda0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3ecbfff640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009d90 tx=0x7f3ec4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4040650 con 0x7f3ecc15bda0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ec402ce90 con 0x7f3ecc15bda0 2024-03-24T10:34:12.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc1e2ef0 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4040a10 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.548+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1e18f0 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.549+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ecc04f4e0 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.549+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ec4040b70 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.550+0000 7f3eb3fff640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 0x7f3eb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.550+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3ec40acd50 con 0x7f3ecc15bda0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.550+0000 7f3ecb7fe640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 0x7f3eb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:12.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.550+0000 7f3ecb7fe640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 0x7f3eb806dcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005880 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:12.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.554+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ec40b0030 con 0x7f3ecc15bda0 2024-03-24T10:34:12.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.778+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3ecc15b2e0 con 0x7f3ecc15bda0 2024-03-24T10:34:12.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.778+0000 7f3eb3fff640 1 -- 172.21.15.1:0/2646611257 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f3ec4077a00 con 0x7f3ecc15bda0 2024-03-24T10:34:12.779 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.780+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 msgr2=0x7f3eb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 0x7f3eb806dcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005880 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 msgr2=0x7f3ecc0a9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009d90 tx=0x7f3ec4004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 shutdown_connections 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eb806b810 0x7f3eb806dcd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc15bda0 0x7f3ecc0a9d30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ecc0cff00 0x7f3ecc0a97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 --2- 172.21.15.1:0/2646611257 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ecc06c910 0x7f3ecc153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:12.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 >> 172.21.15.1:0/2646611257 conn(0x7f3ecc0b17a0 msgr2=0x7f3ecc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:12.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 shutdown_connections 2024-03-24T10:34:12.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:12.781+0000 7f3ed1bc2640 1 -- 172.21.15.1:0/2646611257 wait complete. 2024-03-24T10:34:12.791 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:34:12.792 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-24T10:34:12.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:12.792 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:34:12.807 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:34:12.808 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42688sec preferred_lft 42688sec 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.809 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:34:12.810 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:12.810 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link tentative 2024-03-24T10:34:12.810 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:12.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:12.810 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:34:12.873 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:34:12.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:12.875 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:34:12.931 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:34:12.931 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:34:12.931 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:34:12.931 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:34:12.931 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:34:12.931 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:12.932 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:12.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:34:12.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:34:12.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:34:12.932 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:13.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:13.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:13.067 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:34:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:34:13.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:13.083 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-24T10:34:13.136 INFO:teuthology.orchestra.run.smithi001.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-24T10:34:13.137 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:13.137 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:13.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:34:13.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-24T10:34:13.137 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:13.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:13.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:13.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:13.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 82:41:7c:8d:3e:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::8041:7cff:fe8d:3ee1/64 scope link 2024-03-24T10:34:13.285 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:13.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T10:34:13.353 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:13.353 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:13.353 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:13.353 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.353 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 82:41:7c:8d:3e:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::8041:7cff:fe8d:3ee1/64 scope link 2024-03-24T10:34:13.354 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:13.355 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-24T10:34:13.355 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-24T10:34:13.356 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:13.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:13.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:13.519 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:13.519 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:34:13.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.1 up 2024-03-24T10:34:13.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T10:34:13.519 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:34:13.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:13.600 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-24T10:34:13.601 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:13.601 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:13.656 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-24T10:34:13.656 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:13.656 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:13.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:34:13.779 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:34:13.836 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:34:13.837 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:34:13.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:13.838 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:34:13.891 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:34:13.891 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-24T10:34:13.891 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-24T10:34:13.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:34:13.962 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:14.017 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:34:14.017 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:34:14.017 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:34:14.017 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:14.018 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:14.019 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:34:14.020 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:34:14.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:14.020 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:34:14.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:2024-03-24T10:34:14.020+0000 7f44a3040480 -1 init, newargv = 0x55d023da3930 newargc=15 2024-03-24T10:34:14.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:ceph-fuse[46983]: starting ceph client 2024-03-24T10:34:14.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:ceph-fuse[46983]: starting fuse 2024-03-24T10:34:14.035 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:34:14.036 INFO:teuthology.orchestra.run.smithi001.stdout:44 2024-03-24T10:34:14.036 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:34:14.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.2")) 2024-03-24T10:34:14.038 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:34:14.157 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.2.46983.asok 2024-03-24T10:34:14.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:14.167 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.46983.asok 2024-03-24T10:34:14.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:14.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.46983.asok status 2024-03-24T10:34:14.305 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:34:14.305 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:34:14.305 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:34:14.305 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "2", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "46983", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4743, 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4743 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:14.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.2:0", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 608239560 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.2:0", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 608239560 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4743 192.168.0.2:0/608239560", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.2:0/608239560", 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 8, 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 21, 2024-03-24T10:34:14.307 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:34:14.308 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:34:14.308 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:34:14.308 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:34:14.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:34:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.416+0000 7f49fd664640 1 Processor -- start 2024-03-24T10:34:14.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.416+0000 7f49fd664640 1 -- start start 2024-03-24T10:34:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49fd664640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f815bda0 0x7f49f815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49fd664640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f49f806c910 0x7f49f80cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49fd664640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49f815d3c0 con 0x7f49f80cff00 2024-03-24T10:34:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49fd664640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f49f80b2db0 con 0x7f49f815bda0 2024-03-24T10:34:14.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49fd664640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f49f806e140 con 0x7f49f806c910 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f815bda0 0x7f49f815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f815bda0 0x7f49f815ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58646/0 (socket says 172.21.15.1:58646) 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 -- 172.21.15.1:0/3467512938 learned_addr learned my addr 172.21.15.1:0/3467512938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f67fc640 1 --2- 172.21.15.1:0/3467512938 >> v2:172.21.15.3:3300/0 conn(0x7f49f806c910 0x7f49f80cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 -- 172.21.15.1:0/3467512938 >> v2:172.21.15.3:3300/0 conn(0x7f49f806c910 msgr2=0x7f49f80cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 --2- 172.21.15.1:0/3467512938 >> v2:172.21.15.3:3300/0 conn(0x7f49f806c910 0x7f49f80cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 -- 172.21.15.1:0/3467512938 >> v1:172.21.15.3:6789/0 conn(0x7f49f80cff00 legacy=0x7f49f8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.417+0000 7f49f6ffd640 1 -- 172.21.15.1:0/3467512938 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f806ce40 con 0x7f49f815bda0 2024-03-24T10:34:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49f6ffd640 1 --2- 172.21.15.1:0/3467512938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f815ce80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49f80af5c0 tx=0x7f49e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92e41b99303a6d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49f5ffb640 1 -- 172.21.15.1:0/3467512938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8031870 con 0x7f49f815bda0 2024-03-24T10:34:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49f5ffb640 1 -- 172.21.15.1:0/3467512938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e80319d0 con 0x7f49f815bda0 2024-03-24T10:34:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49f5ffb640 1 -- 172.21.15.1:0/3467512938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8031d60 con 0x7f49f815bda0 2024-03-24T10:34:14.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49fd664640 1 -- 172.21.15.1:0/3467512938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 msgr2=0x7f49f815ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.418+0000 7f49fd664640 1 --2- 172.21.15.1:0/3467512938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f815ce80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49f80af5c0 tx=0x7f49e802b910 comp rx=0 tx=0).stop 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 -- 172.21.15.1:0/3467512938 shutdown_connections 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 --2- 172.21.15.1:0/3467512938 >> v2:172.21.15.3:3300/0 conn(0x7f49f806c910 0x7f49f80cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 --2- 172.21.15.1:0/3467512938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f815ce80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 -- 172.21.15.1:0/3467512938 >> 172.21.15.1:0/3467512938 conn(0x7f49f80b17a0 msgr2=0x7f49f80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 -- 172.21.15.1:0/3467512938 shutdown_connections 2024-03-24T10:34:14.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.419+0000 7f49fd664640 1 -- 172.21.15.1:0/3467512938 wait complete. 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.420+0000 7f49fd664640 1 Processor -- start 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.420+0000 7f49fd664640 1 -- start start 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f806c910 0x7f49f8144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f49f80cff00 0x7f49f80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81e2420 con 0x7f49f815bda0 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81e2700 con 0x7f49f806c910 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f49f81e29e0 con 0x7f49f80cff00 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f806c910 0x7f49f8144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58650/0 (socket says 172.21.15.1:58650) 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 -- 172.21.15.1:0/2527337117 learned_addr learned my addr 172.21.15.1:0/2527337117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f67fc640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f49f80cff00 0x7f49f80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 -- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f49f80cff00 msgr2=0x7f49f80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f49f80cff00 0x7f49f80b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 -- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f806c910 msgr2=0x7f49f8144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f806c910 0x7f49f8144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 -- 172.21.15.1:0/2527337117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e80094d0 con 0x7f49f815bda0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49f77fe640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c7a0 tx=0x7f49f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0003b70 con 0x7f49f815bda0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f0003cf0 con 0x7f49f815bda0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0003ea0 con 0x7f49f815bda0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f81e2cc0 con 0x7f49f815bda0 2024-03-24T10:34:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.421+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f8144a70 con 0x7f49f815bda0 2024-03-24T10:34:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.422+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49f81e31e0 con 0x7f49f815bda0 2024-03-24T10:34:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.423+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f49f000aa90 con 0x7f49f815bda0 2024-03-24T10:34:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.423+0000 7f49df7fe640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 0x7f49e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.423+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49f008a030 con 0x7f49f815bda0 2024-03-24T10:34:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.423+0000 7f49f6ffd640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 0x7f49e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:14.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.424+0000 7f49f6ffd640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 0x7f49e406dcd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49e8036000 tx=0x7f49e803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.428+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f49f81e31e0 con 0x7f49f815bda0 2024-03-24T10:34:14.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.647+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f49f81e31e0 con 0x7f49f815bda0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.647+0000 7f49df7fe640 1 -- 172.21.15.1:0/2527337117 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f49f81e31e0 con 0x7f49f815bda0 2024-03-24T10:34:14.648 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 msgr2=0x7f49e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 0x7f49e406dcd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49e8036000 tx=0x7f49e803c040 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 msgr2=0x7f49f80b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c7a0 tx=0x7f49f000cc60 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 shutdown_connections 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f49e406b810 0x7f49e406dcd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f815bda0 0x7f49f80b3440 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f49f80cff00 0x7f49f80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 --2- 172.21.15.1:0/2527337117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f806c910 0x7f49f8144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 >> 172.21.15.1:0/2527337117 conn(0x7f49f80b17a0 msgr2=0x7f49f80b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 shutdown_connections 2024-03-24T10:34:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:14.649+0000 7f49fd664640 1 -- 172.21.15.1:0/2527337117 wait complete. 2024-03-24T10:34:14.660 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:34:14.660 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:34:14.660 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-03-24T10:34:14.660 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T10:34:14.660 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:14.661 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:34:14.661 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:34:14.661 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:34:14.661 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T10:34:14.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:14.661 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-03-24T10:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42716sec preferred_lft 42716sec 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:14.678 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:14.679 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:14.679 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-03-24T10:34:14.679 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add name ceph-brx type bridge 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr flush dev ceph-brx 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set ceph-brx up 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T10:34:14.679 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:14.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:14.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:14.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:14.887 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:34:14.917 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-24T10:34:14.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:14.919 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-03-24T10:34:14.990 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-03-24T10:34:14.990 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:34:14.990 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:34:14.990 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:34:14.990 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:34:14.991 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:15.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:15.319 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-03-24T10:34:15.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:15.334 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2024-03-24T10:34:15.389 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:15.390 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:15.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:15.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T10:34:15.390 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:15.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:15.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:15.524 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T10:34:15.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T10:34:15.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T10:34:15.525 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:15.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:15.667 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:15.667 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:15.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2024-03-24T10:34:15.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T10:34:15.667 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:15.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:15.750 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-24T10:34:15.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:15.750 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:15.777 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T10:34:15.777 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:15.777 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:15.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2024-03-24T10:34:15.900 DEBUG:teuthology.orchestra.run.smithi003:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:15.956 INFO:teuthology.orchestra.run.smithi003.stdout:/proc 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/dev 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/security 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/shm 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/pts 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/run 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/cgroup 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/pstore 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/bpf 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/ 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/selinux 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/mqueue 2024-03-24T10:34:15.957 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/debug 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/hugepages 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/tracing 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/config 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/run/user/1000 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T10:34:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T10:34:15.959 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns 2024-03-24T10:34:15.959 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:15.959 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:15.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:15.959 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2024-03-24T10:34:16.011 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:34:16.011 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-24T10:34:16.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2024-03-24T10:34:16.084 DEBUG:teuthology.orchestra.run.smithi003:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:16.141 INFO:teuthology.orchestra.run.smithi003.stdout:/proc 2024-03-24T10:34:16.141 INFO:teuthology.orchestra.run.smithi003.stdout:/sys 2024-03-24T10:34:16.141 INFO:teuthology.orchestra.run.smithi003.stdout:/dev 2024-03-24T10:34:16.141 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/security 2024-03-24T10:34:16.141 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/shm 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/pts 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/run 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/cgroup 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/pstore 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/bpf 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/ 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/selinux 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/mqueue 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/debug 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/hugepages 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/tracing 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:16.142 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/config 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/run/user/1000 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:16.143 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:16.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:16.144 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2024-03-24T10:34:16.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi003.stderr:2024-03-24T10:34:16.143+0000 7f67cea40480 -1 init, newargv = 0x55a6ae73c800 newargc=15 2024-03-24T10:34:16.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi003.stderr:ceph-fuse[43895]: starting ceph client 2024-03-24T10:34:16.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi003.stderr:ceph-fuse[43895]: starting fuse 2024-03-24T10:34:16.162 INFO:teuthology.orchestra.run.smithi003.stdout:41 2024-03-24T10:34:16.163 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> import glob 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> import re 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> import os 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> import subprocess 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> def _find_admin_socket(client_name): 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> files = glob.glob(asok_path) 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:16.163 DEBUG:teuthology.orchestra.run.smithi003:> # Given a non-glob path, it better be there 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> if "*" not in asok_path: 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> assert(len(files) == 1) 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> return files[0] 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> for f in files: 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> contents = proc_f.read() 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> if mountpoint in contents: 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> return f 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> print(_find_admin_socket("client.1")) 2024-03-24T10:34:16.164 DEBUG:teuthology.orchestra.run.smithi003:> ' 2024-03-24T10:34:16.283 INFO:teuthology.orchestra.run.smithi003.stdout:/var/run/ceph/ceph-client.1.43895.asok 2024-03-24T10:34:16.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:16.294 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43895.asok 2024-03-24T10:34:16.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:16.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43895.asok status 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": { 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout: "entity_id": "1", 2024-03-24T10:34:16.432 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": "smithi003", 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pid": "43895", 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "root": "/" 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_count": 0, 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_pinned_count": 0, 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 4544, 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "inst": { 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "name": { 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "client", 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 4544 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-03-24T10:34:16.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 888965630 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 888965630 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "inst_str": "client.4544 192.168.0.1:0/888965630", 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr_str": "192.168.0.1:0/888965630", 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "inode_count": 1, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_epoch": 8, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch": 21, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:34:16.434 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklisted": false, 2024-03-24T10:34:16.435 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs" 2024-03-24T10:34:16.435 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-24T10:34:16.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:34:16.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.554+0000 7f53deaf4640 1 Processor -- start 2024-03-24T10:34:16.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.554+0000 7f53deaf4640 1 -- start start 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53deaf4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53d806d2d0 0x7f53d815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53deaf4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d80d10e0 con 0x7f53d80b74c0 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53deaf4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d80b4c70 con 0x7f53d806c770 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53deaf4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f53d80d1340 con 0x7f53d806d2d0 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53dd091640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53d806d2d0 0x7f53d815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53dd091640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53d806d2d0 0x7f53d815d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58660/0 (socket says 172.21.15.1:58660) 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53dd091640 1 -- 172.21.15.1:0/640877958 learned_addr learned my addr 172.21.15.1:0/640877958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:16.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.555+0000 7f53dc890640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53d80b74c0 0x7f53d80d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59638/0 (socket says 172.21.15.1:59638) 2024-03-24T10:34:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53dd091640 1 -- 172.21.15.1:0/640877958 >> v1:172.21.15.3:6789/0 conn(0x7f53d806c770 legacy=0x7f53d806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53d77fe640 1 -- 172.21.15.1:0/640877958 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788142686 0 0) 0x7f53d80d10e0 con 0x7f53d80b74c0 2024-03-24T10:34:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53dd091640 1 -- 172.21.15.1:0/640877958 >> v1:172.21.15.1:6789/0 conn(0x7f53d80b74c0 legacy=0x7f53d80d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53dd091640 1 -- 172.21.15.1:0/640877958 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d80b4f50 con 0x7f53d806d2d0 2024-03-24T10:34:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53dd091640 1 --2- 172.21.15.1:0/640877958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815d8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f53c80097f0 tx=0x7f53c802b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eada4db999057d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.556+0000 7f53d77fe640 1 -- 172.21.15.1:0/640877958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031760 con 0x7f53d806d2d0 2024-03-24T10:34:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53d77fe640 1 -- 172.21.15.1:0/640877958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c80318c0 con 0x7f53d806d2d0 2024-03-24T10:34:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53d77fe640 1 -- 172.21.15.1:0/640877958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031c50 con 0x7f53d806d2d0 2024-03-24T10:34:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 -- 172.21.15.1:0/640877958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 msgr2=0x7f53d815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 --2- 172.21.15.1:0/640877958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815d8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f53c80097f0 tx=0x7f53c802b800 comp rx=0 tx=0).stop 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 -- 172.21.15.1:0/640877958 shutdown_connections 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 --2- 172.21.15.1:0/640877958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815d8f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 -- 172.21.15.1:0/640877958 >> 172.21.15.1:0/640877958 conn(0x7f53d80d1c80 msgr2=0x7f53d80d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 -- 172.21.15.1:0/640877958 shutdown_connections 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.557+0000 7f53deaf4640 1 -- 172.21.15.1:0/640877958 wait complete. 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.558+0000 7f53deaf4640 1 Processor -- start 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.558+0000 7f53deaf4640 1 -- start start 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d806c770 0x7f53d81f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53d80b74c0 0x7f53d815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f53d81595b0 con 0x7f53d806d2d0 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f53d81f7350 con 0x7f53d806c770 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53deaf4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f53d815c610 con 0x7f53d80b74c0 2024-03-24T10:34:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53d7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58670/0 (socket says 172.21.15.1:58670) 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53d7fff640 1 -- 172.21.15.1:0/2452076452 learned_addr learned my addr 172.21.15.1:0/2452076452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.559+0000 7f53dc890640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d806c770 0x7f53d81f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53dd091640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53d80b74c0 0x7f53d815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 -- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53d80b74c0 msgr2=0x7f53d815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53d80b74c0 0x7f53d815aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 -- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d806c770 msgr2=0x7f53d81f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d806c770 0x7f53d81f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 -- 172.21.15.1:0/2452076452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c80094d0 con 0x7f53d806d2d0 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d7fff640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00c7b0 tx=0x7f53cc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc005270 con 0x7f53d806d2d0 2024-03-24T10:34:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53cc0053d0 con 0x7f53d806d2d0 2024-03-24T10:34:16.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc003830 con 0x7f53d806d2d0 2024-03-24T10:34:16.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d8158930 con 0x7f53d806d2d0 2024-03-24T10:34:16.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.560+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d8158ef0 con 0x7f53d806d2d0 2024-03-24T10:34:16.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.561+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d8068680 con 0x7f53d806d2d0 2024-03-24T10:34:16.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.562+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f53cc003990 con 0x7f53d806d2d0 2024-03-24T10:34:16.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.562+0000 7f53d57fa640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 0x7f53c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:16.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.562+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f53cc089730 con 0x7f53d806d2d0 2024-03-24T10:34:16.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.562+0000 7f53dc890640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 0x7f53c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:16.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.563+0000 7f53dc890640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 0x7f53c406dcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001920 tx=0x7f53d00018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:16.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.567+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f53cc054350 con 0x7f53d806d2d0 2024-03-24T10:34:16.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.786+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f53d81596f0 con 0x7f53d806d2d0 2024-03-24T10:34:16.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.786+0000 7f53d57fa640 1 -- 172.21.15.1:0/2452076452 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f53d81596f0 con 0x7f53d806d2d0 2024-03-24T10:34:16.786 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 msgr2=0x7f53c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 0x7f53c406dcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001920 tx=0x7f53d00018b0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 msgr2=0x7f53d815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00c7b0 tx=0x7f53cc00cc70 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 shutdown_connections 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f53c406b810 0x7f53c406dcd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53d80b74c0 0x7f53d815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53d806d2d0 0x7f53d815a660 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 --2- 172.21.15.1:0/2452076452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d806c770 0x7f53d81f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:16.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 >> 172.21.15.1:0/2452076452 conn(0x7f53d80d1c80 msgr2=0x7f53d80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 shutdown_connections 2024-03-24T10:34:16.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:16.788+0000 7f53deaf4640 1 -- 172.21.15.1:0/2452076452 wait complete. 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:34:16.799 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-24T10:34:16.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:16.799 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:16.815 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 42714sec preferred_lft 42714sec 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:16.816 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:10:6e:56:4e:cf brd ff:ff:ff:ff:ff:ff 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::8051:e9ff:fea3:1609/64 scope link tentative 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether ee:10:6e:56:4e:cf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec10:6eff:fe56:4ecf/64 scope link tentative 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:16.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:16.817 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:34:16.881 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-03-24T10:34:16.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:16.883 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-03-24T10:34:16.939 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-03-24T10:34:16.939 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:34:16.939 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:34:16.939 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:34:16.939 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:34:16.939 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:16.939 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:16.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:34:16.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:34:16.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:34:16.940 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:17.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:17.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:17.086 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-03-24T10:34:17.101 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T10:34:17.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:17.102 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2024-03-24T10:34:17.156 INFO:teuthology.orchestra.run.smithi003.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-24T10:34:17.157 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:17.157 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:17.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:34:17.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-24T10:34:17.157 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:17.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:17.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:17.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 22:b6:d0:71:e8:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T10:34:17.308 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T10:34:17.309 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.309 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::20b6:d0ff:fe71:e8f9/64 scope link 2024-03-24T10:34:17.309 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:17.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 22:b6:d0:71:e8:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T10:34:17.377 INFO:teuthology.orchestra.run.smithi003.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T10:34:17.378 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.378 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::20b6:d0ff:fe71:e8f9/64 scope link 2024-03-24T10:34:17.378 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:34:17.379 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-24T10:34:17.379 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:17.544 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:34:17.544 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-03-24T10:34:17.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.1 up 2024-03-24T10:34:17.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T10:34:17.545 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-03-24T10:34:17.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:34:17.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:17.626 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-24T10:34:17.627 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:17.627 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:17.642 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-24T10:34:17.642 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:34:17.642 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:17.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2024-03-24T10:34:17.765 DEBUG:teuthology.orchestra.run.smithi003:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/proc 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/dev 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/security 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/shm 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/pts 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/run 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/cgroup 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/pstore 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/bpf 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/ 2024-03-24T10:34:17.826 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/selinux 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/mqueue 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/debug 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/hugepages 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/tracing 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/config 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/run/user/1000 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T10:34:17.827 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:34:17.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:17.828 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2024-03-24T10:34:17.881 INFO:teuthology.orchestra.run.smithi003.stdout:41 2024-03-24T10:34:17.882 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-24T10:34:17.882 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-24T10:34:17.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2024-03-24T10:34:17.956 DEBUG:teuthology.orchestra.run.smithi003:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:34:18.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi003.stderr:2024-03-24T10:34:18.009+0000 7f34d8a3b480 -1 init, newargv = 0x55d42e885870 newargc=15 2024-03-24T10:34:18.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi003.stderr:ceph-fuse[44352]: starting ceph client 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/proc 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/sys 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/dev 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/security 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/shm 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/pts 2024-03-24T10:34:18.012 INFO:teuthology.orchestra.run.smithi003.stdout:/run 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/cgroup 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/pstore 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/bpf 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/ 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/selinux 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/mqueue 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/debug 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/dev/hugepages 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/tracing 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/fs/fuse/connections 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/sys/kernel/config 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:34:18.013 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/user/1000 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run.smithi003.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:34:18.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.015 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2024-03-24T10:34:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi003.stderr:ceph-fuse[44352]: starting fuse 2024-03-24T10:34:18.029 INFO:teuthology.orchestra.run.smithi003.stdout:41 2024-03-24T10:34:18.029 INFO:teuthology.orchestra.run.smithi003.stdout:44 2024-03-24T10:34:18.029 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> import glob 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> import re 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> import os 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> import subprocess 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> def _find_admin_socket(client_name): 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> files = glob.glob(asok_path) 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> # Given a non-glob path, it better be there 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> if "*" not in asok_path: 2024-03-24T10:34:18.030 DEBUG:teuthology.orchestra.run.smithi003:> assert(len(files) == 1) 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> return files[0] 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> for f in files: 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> contents = proc_f.read() 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> if mountpoint in contents: 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> return f 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> print(_find_admin_socket("client.3")) 2024-03-24T10:34:18.031 DEBUG:teuthology.orchestra.run.smithi003:> ' 2024-03-24T10:34:18.151 INFO:teuthology.orchestra.run.smithi003.stdout:/var/run/ceph/ceph-client.3.44352.asok 2024-03-24T10:34:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-24T10:34:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:34:18.162 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44352.asok 2024-03-24T10:34:18.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44352.asok status 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": { 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "entity_id": "3", 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": "smithi003", 2024-03-24T10:34:18.305 INFO:teuthology.orchestra.run.smithi003.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "pid": "44352", 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "root": "/" 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_count": 0, 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_pinned_count": 0, 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 4553, 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "inst": { 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "name": { 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "client", 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 4553 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-03-24T10:34:18.306 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "192.168.0.2:0", 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 792866111 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "192.168.0.2:0", 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 792866111 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "inst_str": "client.4553 192.168.0.2:0/792866111", 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "addr_str": "192.168.0.2:0/792866111", 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "inode_count": 1, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_epoch": 8, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch": 21, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:34:18.307 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklisted": false, 2024-03-24T10:34:18.308 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs" 2024-03-24T10:34:18.308 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-24T10:34:18.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.314 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:34:18.314 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:18.333 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:34:18.333 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:18.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:18.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.406 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:34:18.406 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:18.464 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:34:18.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:18.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:18.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.538 DEBUG:teuthology.orchestra.run.smithi003:> stat --file-system '--printf=%T 2024-03-24T10:34:18.538 DEBUG:teuthology.orchestra.run.smithi003:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:18.565 INFO:teuthology.orchestra.run.smithi003.stdout:fuseblk 2024-03-24T10:34:18.565 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:18.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:18.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.633 DEBUG:teuthology.orchestra.run.smithi003:> stat --file-system '--printf=%T 2024-03-24T10:34:18.633 DEBUG:teuthology.orchestra.run.smithi003:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:18.688 INFO:teuthology.orchestra.run.smithi003.stdout:fuseblk 2024-03-24T10:34:18.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:18.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:18.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:18.756 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-24T10:34:18.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:18.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 Processor -- start 2024-03-24T10:34:18.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 -- start start 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ebc0d2bb0 0x7f3ebc0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d1000 0x7f3ebc0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ebc15e770 con 0x7f3ebc15e390 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3ebc0b3190 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.900+0000 7f3ec274f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ebc0d4e40 con 0x7f3ebc0d1000 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ebc15e390 0x7f3ebc0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59642/0 (socket says 172.21.15.1:59642) 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebbfff640 1 -- 172.21.15.1:0/1958076647 learned_addr learned my addr 172.21.15.1:0/1958076647 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ec0cec640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d1000 0x7f3ebc0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/1958076647 >> v2:172.21.15.3:3300/0 conn(0x7f3ebc0d2bb0 0x7f3ebc0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebaffd640 1 -- 172.21.15.1:0/1958076647 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017330597 0 0) 0x7f3ea4001110 con 0x7f3ebc15e390 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebaffd640 1 -- 172.21.15.1:0/1958076647 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea8003520 con 0x7f3ebc15e390 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebb7fe640 1 -- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d1000 msgr2=0x7f3ebc0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d1000 0x7f3ebc0b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebb7fe640 1 -- 172.21.15.1:0/1958076647 >> v1:172.21.15.1:6789/0 conn(0x7f3ebc15e390 legacy=0x7f3ebc0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.901+0000 7f3ebb7fe640 1 -- 172.21.15.1:0/1958076647 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ebc0d1e40 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b3b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097a0 tx=0x7f3eac02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3fcdea6ed14976 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ebaffd640 1 -- 172.21.15.1:0/1958076647 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac02f050 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ebaffd640 1 -- 172.21.15.1:0/1958076647 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eac031700 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ebaffd640 1 -- 172.21.15.1:0/1958076647 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac031a70 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ec274f640 1 -- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d2bb0 msgr2=0x7f3ebc0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.902+0000 7f3ec274f640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b3b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097a0 tx=0x7f3eac02b7a0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 -- 172.21.15.1:0/1958076647 shutdown_connections 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d1000 0x7f3ebc0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 --2- 172.21.15.1:0/1958076647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b3b90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 -- 172.21.15.1:0/1958076647 >> 172.21.15.1:0/1958076647 conn(0x7f3ebc0d3ec0 msgr2=0x7f3ebc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 -- 172.21.15.1:0/1958076647 shutdown_connections 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.903+0000 7f3ec274f640 1 -- 172.21.15.1:0/1958076647 wait complete. 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.904+0000 7f3ec274f640 1 Processor -- start 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.904+0000 7f3ec274f640 1 -- start start 2024-03-24T10:34:18.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 0x7f3ebc1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc159430 con 0x7f3ebc15e390 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc157870 con 0x7f3ebc0d1000 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec274f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ebc0b5420 con 0x7f3ebc0d2bb0 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec0cec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec0cec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58680/0 (socket says 172.21.15.1:58680) 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec0cec640 1 -- 172.21.15.1:0/965869616 learned_addr learned my addr 172.21.15.1:0/965869616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 0x7f3ebc1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 -- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d2bb0 msgr2=0x7f3ebc0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 -- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 msgr2=0x7f3ebc0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 -- 172.21.15.1:0/965869616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eac009400 con 0x7f3ebc0d1000 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ec0cec640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.905+0000 7f3ebbfff640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 0x7f3ebc1f3990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400c390 tx=0x7f3ea400c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.906+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4005ce0 con 0x7f3ebc0d1000 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.906+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea4005e40 con 0x7f3ebc0d1000 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.906+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ebc0b4b80 con 0x7f3ebc0d1000 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.906+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40072f0 con 0x7f3ebc0d1000 2024-03-24T10:34:18.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.906+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ebc0b5140 con 0x7f3ebc0d1000 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.907+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ebc04f4e0 con 0x7f3ebc0d1000 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.907+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ea4007510 con 0x7f3ebc0d1000 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.907+0000 7f3eb8ff9640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 0x7f3e9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.908+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3ea40894f0 con 0x7f3ebc0d1000 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.908+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 0x7f3e9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.908+0000 7f3ebb7fe640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 0x7f3e9c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3eac035000 tx=0x7f3eac008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:18.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:18.912+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ea4053f20 con 0x7f3ebc0d1000 2024-03-24T10:34:19.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.094+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ebc15e770 con 0x7f3ebc0d1000 2024-03-24T10:34:19.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.096+0000 7f3eb8ff9640 1 -- 172.21.15.1:0/965869616 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f3ebc15e770 con 0x7f3ebc0d1000 2024-03-24T10:34:19.096 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:19.096 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 msgr2=0x7f3e9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 0x7f3e9c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3eac035000 tx=0x7f3eac008030 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 msgr2=0x7f3ebc1f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 0x7f3ebc1f3990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400c390 tx=0x7f3ea400c790 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 shutdown_connections 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3e9c06b810 0x7f3e9c06dcd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ebc15e390 0x7f3ebc0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ebc0d2bb0 0x7f3ebc0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 --2- 172.21.15.1:0/965869616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ebc0d1000 0x7f3ebc1f3990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 >> 172.21.15.1:0/965869616 conn(0x7f3ebc0d3ec0 msgr2=0x7f3ebc0b8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 shutdown_connections 2024-03-24T10:34:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.098+0000 7f3ec274f640 1 -- 172.21.15.1:0/965869616 wait complete. 2024-03-24T10:34:19.109 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:19.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.219+0000 7f0dcbb0c640 1 Processor -- start 2024-03-24T10:34:19.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.219+0000 7f0dcbb0c640 1 -- start start 2024-03-24T10:34:19.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dcbb0c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc4161760 0x7f0dc40b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dcbb0c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0dc40ceb00 0x7f0dc40d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dcbb0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc40cfef0 con 0x7f0dc40d0d00 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dcbb0c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f0dc40d1190 con 0x7f0dc40ceb00 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dcbb0c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc40b5530 con 0x7f0dc4161760 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dca0a9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0dc40d0d00 0x7f0dc4161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59658/0 (socket says 172.21.15.1:59658) 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dca0a9640 1 -- 172.21.15.1:0/3824991393 learned_addr learned my addr 172.21.15.1:0/3824991393 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc98a8640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc4161760 0x7f0dc40b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc90a7640 1 --2- 172.21.15.1:0/3824991393 >> v2:172.21.15.3:3300/0 conn(0x7f0dc40ceb00 0x7f0dc40d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc88a6640 1 -- 172.21.15.1:0/3824991393 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3723900792 0 0) 0x7f0dc40cfef0 con 0x7f0dc40d0d00 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc88a6640 1 -- 172.21.15.1:0/3824991393 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0db0003520 con 0x7f0dc40d0d00 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc90a7640 1 -- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc4161760 msgr2=0x7f0dc40b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc90a7640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc4161760 0x7f0dc40b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc90a7640 1 -- 172.21.15.1:0/3824991393 >> v1:172.21.15.1:6789/0 conn(0x7f0dc40d0d00 legacy=0x7f0dc4161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.220+0000 7f0dc90a7640 1 -- 172.21.15.1:0/3824991393 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc40b6d40 con 0x7f0dc40ceb00 2024-03-24T10:34:19.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.221+0000 7f0dc90a7640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40ceb00 0x7f0dc40d07c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0db40097a0 tx=0x7f0db402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19febc11dddfef85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.221+0000 7f0dc88a6640 1 -- 172.21.15.1:0/3824991393 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db402e070 con 0x7f0dc40ceb00 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.221+0000 7f0dc88a6640 1 -- 172.21.15.1:0/3824991393 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db40315a0 con 0x7f0dc40ceb00 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.221+0000 7f0dc88a6640 1 -- 172.21.15.1:0/3824991393 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4031910 con 0x7f0dc40ceb00 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40ceb00 msgr2=0x7f0dc40d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40ceb00 0x7f0dc40d07c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0db40097a0 tx=0x7f0db402b840 comp rx=0 tx=0).stop 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/3824991393 shutdown_connections 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40ceb00 0x7f0dc40d07c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/3824991393 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc4161760 0x7f0dc40b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/3824991393 >> 172.21.15.1:0/3824991393 conn(0x7f0dc409d1c0 msgr2=0x7f0dc40d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/3824991393 shutdown_connections 2024-03-24T10:34:19.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.222+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/3824991393 wait complete. 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.223+0000 7f0dcbb0c640 1 Processor -- start 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.223+0000 7f0dcbb0c640 1 -- start start 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc40ceb00 0x7f0dc4153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40d0d00 0x7f0dc40cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc41539c0 con 0x7f0dc4161760 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc40c87d0 con 0x7f0dc40d0d00 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc40cbe10 con 0x7f0dc40ceb00 2024-03-24T10:34:19.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dc98a8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc40ceb00 0x7f0dc4153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58692/0 (socket says 172.21.15.1:58692) 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 -- 172.21.15.1:0/1838571102 learned_addr learned my addr 172.21.15.1:0/1838571102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dc90a7640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40d0d00 0x7f0dc40cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 -- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc40ceb00 msgr2=0x7f0dc4153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc40ceb00 0x7f0dc4153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 -- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40d0d00 msgr2=0x7f0dc40cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40d0d00 0x7f0dc40cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 -- 172.21.15.1:0/1838571102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db4009400 con 0x7f0dc4161760 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dca0a9640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00caa0 tx=0x7f0dbc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00aba0 con 0x7f0dc4161760 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dbc00ad00 con 0x7f0dc4161760 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc40c7ea0 con 0x7f0dc4161760 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc012620 con 0x7f0dc4161760 2024-03-24T10:34:19.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.224+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc40c8460 con 0x7f0dc4161760 2024-03-24T10:34:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.226+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc404f4e0 con 0x7f0dc4161760 2024-03-24T10:34:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.226+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0dbc007070 con 0x7f0dc4161760 2024-03-24T10:34:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.226+0000 7f0dba7fc640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 0x7f0da806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.226+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0dbc08a080 con 0x7f0dc4161760 2024-03-24T10:34:19.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.226+0000 7f0dc98a8640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 0x7f0da806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.227+0000 7f0dc98a8640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 0x7f0da806dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009aa0 tx=0x7f0dc0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.231+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0dbc054be0 con 0x7f0dc4161760 2024-03-24T10:34:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.411+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0dc40c8910 con 0x7f0dc4161760 2024-03-24T10:34:19.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.411+0000 7f0dba7fc640 1 -- 172.21.15.1:0/1838571102 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f0dbc0582a0 con 0x7f0dc4161760 2024-03-24T10:34:19.411 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 msgr2=0x7f0da806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 0x7f0da806dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009aa0 tx=0x7f0dc0009190 comp rx=0 tx=0).stop 2024-03-24T10:34:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 msgr2=0x7f0dc40ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00caa0 tx=0x7f0dbc00cf60 comp rx=0 tx=0).stop 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 shutdown_connections 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da806b810 0x7f0da806dcd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dc4161760 0x7f0dc40ccd50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dc40d0d00 0x7f0dc40cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 --2- 172.21.15.1:0/1838571102 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dc40ceb00 0x7f0dc4153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 >> 172.21.15.1:0/1838571102 conn(0x7f0dc409d1c0 msgr2=0x7f0dc40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 shutdown_connections 2024-03-24T10:34:19.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.413+0000 7f0dcbb0c640 1 -- 172.21.15.1:0/1838571102 wait complete. 2024-03-24T10:34:19.424 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.528+0000 7feaa1b93640 1 Processor -- start 2024-03-24T10:34:19.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.528+0000 7feaa1b93640 1 -- start start 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7feaa1b93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7feaa1b93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea9c0cd220 0x7fea9c0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7feaa1b93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9c0d0b50 con 0x7fea9c0d0720 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7feaa1b93640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c0cfd90 con 0x7fea9c0cd220 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7feaa1b93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c0cdac0 con 0x7fea9c15bfa0 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea9c0cd220 0x7fea9c0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fea9c0d0720 0x7fea9c0ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59660/0 (socket says 172.21.15.1:59660) 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9bfff640 1 -- 172.21.15.1:0/418098917 learned_addr learned my addr 172.21.15.1:0/418098917 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 --2- 172.21.15.1:0/418098917 >> v2:172.21.15.1:3300/0 conn(0x7fea9c0cd220 0x7fea9c0d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58706/0 (socket says 172.21.15.1:58706) 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 -- 172.21.15.1:0/418098917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 msgr2=0x7fea9c0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:19.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 -- 172.21.15.1:0/418098917 >> v1:172.21.15.1:6789/0 conn(0x7fea9c0d0720 legacy=0x7fea9c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9a7fc640 1 -- 172.21.15.1:0/418098917 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64306961 0 0) 0x7fea9c0d0b50 con 0x7fea9c0d0720 2024-03-24T10:34:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.529+0000 7fea9affd640 1 -- 172.21.15.1:0/418098917 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0cd7e0 con 0x7fea9c0cd220 2024-03-24T10:34:19.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.530+0000 7fea9b7fe640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c0d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.530+0000 7fea9affd640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c0d01e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea900097a0 tx=0x7fea9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01bc3c77d392489 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.530+0000 7fea9a7fc640 1 -- 172.21.15.1:0/418098917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90031830 con 0x7fea9c0cd220 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.530+0000 7fea9a7fc640 1 -- 172.21.15.1:0/418098917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea90031990 con 0x7fea9c0cd220 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.530+0000 7fea9a7fc640 1 -- 172.21.15.1:0/418098917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90031d20 con 0x7fea9c0cd220 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 -- 172.21.15.1:0/418098917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 msgr2=0x7fea9c0d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c0d01e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea900097a0 tx=0x7fea9002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 -- 172.21.15.1:0/418098917 shutdown_connections 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c0d01e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 --2- 172.21.15.1:0/418098917 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c0d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 -- 172.21.15.1:0/418098917 >> 172.21.15.1:0/418098917 conn(0x7fea9c0d1c00 msgr2=0x7fea9c0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 -- 172.21.15.1:0/418098917 shutdown_connections 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.531+0000 7feaa1b93640 1 -- 172.21.15.1:0/418098917 wait complete. 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.532+0000 7feaa1b93640 1 Processor -- start 2024-03-24T10:34:19.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.532+0000 7feaa1b93640 1 -- start start 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 0x7fea9c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c1f1340 con 0x7fea9c0cd220 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c152010 con 0x7fea9c0d0720 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7feaa1b93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c15d430 con 0x7fea9c15bfa0 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58708/0 (socket says 172.21.15.1:58708) 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 -- 172.21.15.1:0/3578901212 learned_addr learned my addr 172.21.15.1:0/3578901212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9affd640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 0x7fea9c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9bfff640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 -- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 msgr2=0x7fea9c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 -- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 msgr2=0x7fea9c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.533+0000 7fea9b7fe640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 0x7fea9c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea9b7fe640 1 -- 172.21.15.1:0/3578901212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c009400 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea9affd640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 0x7fea9c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea9b7fe640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e960 tx=0x7fea8c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00ca00 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c00cb60 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c013470 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea90009400 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.534+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c1525b0 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.535+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c152740 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.535+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea8c0135d0 con 0x7fea9c0cd220 2024-03-24T10:34:19.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.535+0000 7fea83fff640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 0x7fea8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.536+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fea8c08a390 con 0x7fea9c0cd220 2024-03-24T10:34:19.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.536+0000 7fea9affd640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 0x7fea8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.536+0000 7fea9affd640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 0x7fea8806dcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea90036000 tx=0x7fea9003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.540+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea8c054e60 con 0x7fea9c0cd220 2024-03-24T10:34:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.723+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea9c04f470 con 0x7fea9c0cd220 2024-03-24T10:34:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.723+0000 7fea83fff640 1 -- 172.21.15.1:0/3578901212 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fea8c058520 con 0x7fea9c0cd220 2024-03-24T10:34:19.724 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 msgr2=0x7fea8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 0x7fea8806dcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fea90036000 tx=0x7fea9003c040 comp rx=0 tx=0).stop 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 msgr2=0x7fea9c1f0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e960 tx=0x7fea8c00ee20 comp rx=0 tx=0).stop 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 shutdown_connections 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fea8806b810 0x7fea8806dcd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fea9c15bfa0 0x7fea9c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea9c0d0720 0x7fea9c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 --2- 172.21.15.1:0/3578901212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea9c0cd220 0x7fea9c1f0e00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 >> 172.21.15.1:0/3578901212 conn(0x7fea9c0d1c00 msgr2=0x7fea9c06eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 shutdown_connections 2024-03-24T10:34:19.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.725+0000 7feaa1b93640 1 -- 172.21.15.1:0/3578901212 wait complete. 2024-03-24T10:34:19.736 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:19.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.840+0000 7f78ac73e640 1 Processor -- start 2024-03-24T10:34:19.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.840+0000 7f78ac73e640 1 -- start start 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78ac73e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f78a41600e0 0x7f78a40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78ac73e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a40ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78ac73e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a4145b10 con 0x7f78a4145730 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78ac73e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f78a4071be0 con 0x7f78a41600e0 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78ac73e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f78a406d620 con 0x7f78a406d120 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78aa4da640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f78a4145730 0x7f78a415f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59662/0 (socket says 172.21.15.1:59662) 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78aa4da640 1 -- 172.21.15.1:0/2475406843 learned_addr learned my addr 172.21.15.1:0/2475406843 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.841+0000 7f78aacdb640 1 --2- 172.21.15.1:0/2475406843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a40ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78a9cd9640 1 --2- 172.21.15.1:0/2475406843 >> v2:172.21.15.3:3300/0 conn(0x7f78a41600e0 0x7f78a40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78a94d8640 1 -- 172.21.15.1:0/2475406843 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702395673 0 0) 0x7f78a4145b10 con 0x7f78a4145730 2024-03-24T10:34:19.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78a94d8640 1 -- 172.21.15.1:0/2475406843 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7890003520 con 0x7f78a4145730 2024-03-24T10:34:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78aacdb640 1 -- 172.21.15.1:0/2475406843 >> v2:172.21.15.3:3300/0 conn(0x7f78a41600e0 msgr2=0x7f78a40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78aacdb640 1 --2- 172.21.15.1:0/2475406843 >> v2:172.21.15.3:3300/0 conn(0x7f78a41600e0 0x7f78a40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78aacdb640 1 -- 172.21.15.1:0/2475406843 >> v1:172.21.15.1:6789/0 conn(0x7f78a4145730 legacy=0x7f78a415f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.842+0000 7f78aacdb640 1 -- 172.21.15.1:0/2475406843 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a406d900 con 0x7f78a406d120 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78aacdb640 1 --2- 172.21.15.1:0/2475406843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a40ce7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009d90 tx=0x7f78a002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6124b10fdb62ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78a94d8640 1 -- 172.21.15.1:0/2475406843 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a002cc50 con 0x7f78a406d120 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78a94d8640 1 -- 172.21.15.1:0/2475406843 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a002cdb0 con 0x7f78a406d120 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78a94d8640 1 -- 172.21.15.1:0/2475406843 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0034660 con 0x7f78a406d120 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 -- 172.21.15.1:0/2475406843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 msgr2=0x7f78a40ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 --2- 172.21.15.1:0/2475406843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a40ce7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009d90 tx=0x7f78a002bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 -- 172.21.15.1:0/2475406843 shutdown_connections 2024-03-24T10:34:19.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 --2- 172.21.15.1:0/2475406843 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a40ce7a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 --2- 172.21.15.1:0/2475406843 >> v2:172.21.15.3:3300/0 conn(0x7f78a41600e0 0x7f78a40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 -- 172.21.15.1:0/2475406843 >> 172.21.15.1:0/2475406843 conn(0x7f78a40b1710 msgr2=0x7f78a40b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 -- 172.21.15.1:0/2475406843 shutdown_connections 2024-03-24T10:34:19.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.843+0000 7f78ac73e640 1 -- 172.21.15.1:0/2475406843 wait complete. 2024-03-24T10:34:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 Processor -- start 2024-03-24T10:34:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 -- start start 2024-03-24T10:34:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a4139840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 0x7f78a413aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78a413b3e0 con 0x7f78a41600e0 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4138f80 con 0x7f78a4145730 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78ac73e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f78a4139e20 con 0x7f78a406d120 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.845+0000 7f78aacdb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aa4da640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a4139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78a9cd9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 0x7f78a413aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58716/0 (socket says 172.21.15.1:58716) 2024-03-24T10:34:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 -- 172.21.15.1:0/4084490420 learned_addr learned my addr 172.21.15.1:0/4084490420 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 -- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 msgr2=0x7f78a4139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a4139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 -- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 msgr2=0x7f78a413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 0x7f78a413aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 -- 172.21.15.1:0/4084490420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a0009a70 con 0x7f78a41600e0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aa4da640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a4139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78a9cd9640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 0x7f78a413aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78aacdb640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009d90 tx=0x7f78a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0040650 con 0x7f78a41600e0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a00407b0 con 0x7f78a41600e0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a413a100 con 0x7f78a41600e0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0040b60 con 0x7f78a41600e0 2024-03-24T10:34:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.846+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a413a490 con 0x7f78a41600e0 2024-03-24T10:34:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.847+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a413a290 con 0x7f78a41600e0 2024-03-24T10:34:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.848+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f78a0040cc0 con 0x7f78a41600e0 2024-03-24T10:34:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.848+0000 7f789affd640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 0x7f788806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.848+0000 7f78aa4da640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 0x7f788806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.848+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f78a00acd80 con 0x7f78a41600e0 2024-03-24T10:34:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.849+0000 7f78aa4da640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 0x7f788806dcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f789c005880 tx=0x7f789c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:19.853+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78a0077850 con 0x7f78a41600e0 2024-03-24T10:34:20.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.031+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78a4160880 con 0x7f78a41600e0 2024-03-24T10:34:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.031+0000 7f789affd640 1 -- 172.21.15.1:0/4084490420 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f78a0044020 con 0x7f78a41600e0 2024-03-24T10:34:20.032 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 msgr2=0x7f788806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 0x7f788806dcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f789c005880 tx=0x7f789c009040 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 msgr2=0x7f78a4138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009d90 tx=0x7f78a0004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 shutdown_connections 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f788806b810 0x7f788806dcd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78a41600e0 0x7f78a4138a40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78a4145730 0x7f78a413aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 --2- 172.21.15.1:0/4084490420 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f78a406d120 0x7f78a4139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 >> 172.21.15.1:0/4084490420 conn(0x7f78a40b1710 msgr2=0x7f78a4143590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 shutdown_connections 2024-03-24T10:34:20.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.033+0000 7f78ac73e640 1 -- 172.21.15.1:0/4084490420 wait complete. 2024-03-24T10:34:20.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.145+0000 7f4dbc50c640 1 Processor -- start 2024-03-24T10:34:20.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.145+0000 7f4dbc50c640 1 -- start start 2024-03-24T10:34:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.146+0000 7f4dbc50c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4db40d1000 0x7f4db40b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.146+0000 7f4dbc50c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db415e770 con 0x7f4db415e390 2024-03-24T10:34:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.146+0000 7f4dbc50c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db40d4e40 con 0x7f4db40d2bb0 2024-03-24T10:34:20.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.146+0000 7f4dbc50c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4db40b3190 con 0x7f4db40d1000 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dba2a8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4db415e390 0x7f4db40d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59674/0 (socket says 172.21.15.1:59674) 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dba2a8640 1 -- 172.21.15.1:0/2187258449 learned_addr learned my addr 172.21.15.1:0/2187258449 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dbaaa9640 1 --2- 172.21.15.1:0/2187258449 >> v2:172.21.15.3:3300/0 conn(0x7f4db40d1000 0x7f4db40b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369024937 0 0) 0x7f4da8001110 con 0x7f4db415e390 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da0003520 con 0x7f4db415e390 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592150572 0 0) 0x7f4db40d4e40 con 0x7f4db40d2bb0 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da8001110 con 0x7f4db40d2bb0 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dbaaa9640 1 -- 172.21.15.1:0/2187258449 >> v1:172.21.15.3:6789/0 conn(0x7f4db40d2bb0 legacy=0x7f4db40b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dbaaa9640 1 -- 172.21.15.1:0/2187258449 >> v1:172.21.15.1:6789/0 conn(0x7f4db415e390 legacy=0x7f4db40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.147+0000 7f4dbaaa9640 1 -- 172.21.15.1:0/2187258449 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db40d1e40 con 0x7f4db40d1000 2024-03-24T10:34:20.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbaaa9640 1 --2- 172.21.15.1:0/2187258449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40b42a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009d90 tx=0x7f4dac02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3655c3039795e48b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac02cbd0 con 0x7f4db40d1000 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dac02cd30 con 0x7f4db40d1000 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4db92a6640 1 -- 172.21.15.1:0/2187258449 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac02cea0 con 0x7f4db40d1000 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbc50c640 1 -- 172.21.15.1:0/2187258449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 msgr2=0x7f4db40b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/2187258449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40b42a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009d90 tx=0x7f4dac02be20 comp rx=0 tx=0).stop 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbc50c640 1 -- 172.21.15.1:0/2187258449 shutdown_connections 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/2187258449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40b42a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.148+0000 7f4dbc50c640 1 -- 172.21.15.1:0/2187258449 >> 172.21.15.1:0/2187258449 conn(0x7f4db40d3ec0 msgr2=0x7f4db40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.149+0000 7f4dbc50c640 1 -- 172.21.15.1:0/2187258449 shutdown_connections 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.149+0000 7f4dbc50c640 1 -- 172.21.15.1:0/2187258449 wait complete. 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 Processor -- start 2024-03-24T10:34:20.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 -- start start 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 0x7f4db406d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4db4159fe0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4db4159d00 con 0x7f4db40d1000 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.150+0000 7f4dbc50c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4db40cb0c0 con 0x7f4db415e390 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dba2a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dbaaa9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 0x7f4db406d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58722/0 (socket says 172.21.15.1:58722) 2024-03-24T10:34:20.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 -- 172.21.15.1:0/1978723001 learned_addr learned my addr 172.21.15.1:0/1978723001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:20.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dba2a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:42888/0 (socket says 172.21.15.1:42888) 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 -- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 msgr2=0x7f4db406d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 0x7f4db406d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 -- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 msgr2=0x7f4db40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 -- 172.21.15.1:0/1978723001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac009a70 con 0x7f4db40d2bb0 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dba2a8640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dbaaa9640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 0x7f4db406d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4db9aa7640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c500 tx=0x7f4db000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00052d0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db0005470 con 0x7f4db40d2bb0 2024-03-24T10:34:20.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db40cb3a0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0003830 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.151+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db40cdce0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.152+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db40cb8c0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.153+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4db0003990 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.153+0000 7f4da6ffd640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.153+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4db0089750 con 0x7f4db40d2bb0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.153+0000 7f4dba2a8640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.154+0000 7f4dba2a8640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 0x7f4d9806dd00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4da8000f60 tx=0x7f4da8007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.158+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4db40cb8c0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.339+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4db40cb8c0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.340+0000 7f4da6ffd640 1 -- 172.21.15.1:0/1978723001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f4db40cb8c0 con 0x7f4db40d2bb0 2024-03-24T10:34:20.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:20.340 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 msgr2=0x7f4d9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 0x7f4d9806dd00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4da8000f60 tx=0x7f4da8007d50 comp rx=0 tx=0).stop 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 msgr2=0x7f4db406cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c500 tx=0x7f4db000c9c0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 shutdown_connections 2024-03-24T10:34:20.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4db415e390 0x7f4db406d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4db40d2bb0 0x7f4db406cbf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 --2- 172.21.15.1:0/1978723001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db40d1000 0x7f4db40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 >> 172.21.15.1:0/1978723001 conn(0x7f4db40d3ec0 msgr2=0x7f4db40cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 shutdown_connections 2024-03-24T10:34:20.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.342+0000 7f4dbc50c640 1 -- 172.21.15.1:0/1978723001 wait complete. 2024-03-24T10:34:20.353 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:20.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.457+0000 7f286263b640 1 Processor -- start 2024-03-24T10:34:20.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.457+0000 7f286263b640 1 -- start start 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f286263b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f286263b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285c154fc0 con 0x7f285c0b4290 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f286263b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285c071370 con 0x7f285c0cf450 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f286263b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f285c070bd0 con 0x7f285c154be0 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f2860bd8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f285c0b4290 0x7f285c15e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59682/0 (socket says 172.21.15.1:59682) 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f2860bd8640 1 -- 172.21.15.1:0/4203849737 learned_addr learned my addr 172.21.15.1:0/4203849737 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.458+0000 7f285bfff640 1 --2- 172.21.15.1:0/4203849737 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645561897 0 0) 0x7f285c154fc0 con 0x7f285c0b4290 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2844003520 con 0x7f285c0b4290 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410837387 0 0) 0x7f285c071370 con 0x7f285c0cf450 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f285c154fc0 con 0x7f285c0cf450 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285bfff640 1 -- 172.21.15.1:0/4203849737 >> v1:172.21.15.3:6789/0 conn(0x7f285c0cf450 legacy=0x7f285c0b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285bfff640 1 -- 172.21.15.1:0/4203849737 >> v1:172.21.15.1:6789/0 conn(0x7f285c0b4290 legacy=0x7f285c15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285bfff640 1 -- 172.21.15.1:0/4203849737 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c0b36a0 con 0x7f285c154be0 2024-03-24T10:34:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.459+0000 7f285bfff640 1 --2- 172.21.15.1:0/4203849737 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c06d4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2850009800 tx=0x7f285002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b375868903e5b75 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850031800 con 0x7f285c154be0 2024-03-24T10:34:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2850031960 con 0x7f285c154be0 2024-03-24T10:34:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f285affd640 1 -- 172.21.15.1:0/4203849737 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850031cf0 con 0x7f285c154be0 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 -- 172.21.15.1:0/4203849737 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 msgr2=0x7f285c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 --2- 172.21.15.1:0/4203849737 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c06d4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2850009800 tx=0x7f285002b8a0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 -- 172.21.15.1:0/4203849737 shutdown_connections 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 --2- 172.21.15.1:0/4203849737 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c06d4a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 -- 172.21.15.1:0/4203849737 >> 172.21.15.1:0/4203849737 conn(0x7f285c0b1710 msgr2=0x7f285c0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 -- 172.21.15.1:0/4203849737 shutdown_connections 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.460+0000 7f286263b640 1 -- 172.21.15.1:0/4203849737 wait complete. 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 Processor -- start 2024-03-24T10:34:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 -- start start 2024-03-24T10:34:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f285c0cf450 0x7f285c168970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c1402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f285c137fd0 con 0x7f285c0b4290 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f285c1382b0 con 0x7f285c0cf450 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.462+0000 7f286263b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f285c138590 con 0x7f285c154be0 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58728/0 (socket says 172.21.15.1:58728) 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 -- 172.21.15.1:0/3690611137 learned_addr learned my addr 172.21.15.1:0/3690611137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285b7fe640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f285c0cf450 0x7f285c168970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f2860bd8640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c1402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 -- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 msgr2=0x7f285c1402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c1402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 -- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f285c0cf450 msgr2=0x7f285c168970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f285c0cf450 0x7f285c168970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 -- 172.21.15.1:0/3690611137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2850009400 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f2860bd8640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c1402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f285bfff640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2850000c00 tx=0x7f28500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285003bc70 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285003be10 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c0cca20 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850045630 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.463+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c1407e0 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.464+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285c0cc880 con 0x7f285c0b4290 2024-03-24T10:34:20.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.465+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f285004e450 con 0x7f285c0b4290 2024-03-24T10:34:20.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.465+0000 7f2858ff9640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 0x7f284806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.465+0000 7f285b7fe640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 0x7f284806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.465+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f285002e070 con 0x7f285c0b4290 2024-03-24T10:34:20.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.466+0000 7f285b7fe640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 0x7f284806dd00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f285c153f80 tx=0x7f284c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.470+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f285c0cc880 con 0x7f285c0b4290 2024-03-24T10:34:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.649+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f285c0cc880 con 0x7f285c0b4290 2024-03-24T10:34:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.650+0000 7f2858ff9640 1 -- 172.21.15.1:0/3690611137 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f285c0cc880 con 0x7f285c0b4290 2024-03-24T10:34:20.651 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.651+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 msgr2=0x7f284806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.651+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 0x7f284806dd00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f285c153f80 tx=0x7f284c009040 comp rx=0 tx=0).stop 2024-03-24T10:34:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 msgr2=0x7f285c136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2850000c00 tx=0x7f28500046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 shutdown_connections 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284806b840 0x7f284806dd00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f285c154be0 0x7f285c1402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f285c0cf450 0x7f285c168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 --2- 172.21.15.1:0/3690611137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f285c0b4290 0x7f285c136910 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 >> 172.21.15.1:0/3690611137 conn(0x7f285c0b1710 msgr2=0x7f285c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 shutdown_connections 2024-03-24T10:34:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.652+0000 7f286263b640 1 -- 172.21.15.1:0/3690611137 wait complete. 2024-03-24T10:34:20.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:20.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.766+0000 7f2b27d93640 1 Processor -- start 2024-03-24T10:34:20.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.766+0000 7f2b27d93640 1 -- start start 2024-03-24T10:34:20.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b27d93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b2015bfa0 0x7f2b200d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b27d93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b200cd220 0x7f2b200d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b27d93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b200d0b50 con 0x7f2b200d0720 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b27d93640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2b200cfd90 con 0x7f2b200cd220 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b27d93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2b200cdac0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b2532e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b200cd220 0x7f2b200d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b2532e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2b200cd220 0x7f2b200d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34784/0 (socket says 172.21.15.1:34784) 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b2532e640 1 -- 172.21.15.1:0/2568675188 learned_addr learned my addr 172.21.15.1:0/2568675188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b26330640 1 --1- 172.21.15.1:0/2568675188 >> v1:172.21.15.1:6789/0 conn(0x7f2b200d0720 0x7f2b200ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32960/0 (socket says 172.21.15.1:32960) 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.767+0000 7f2b25b2f640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b2015bfa0 0x7f2b200d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b2532e640 1 -- 172.21.15.1:0/2568675188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b2015bfa0 msgr2=0x7f2b200d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b2532e640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b2015bfa0 0x7f2b200d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b2532e640 1 -- 172.21.15.1:0/2568675188 >> v1:172.21.15.1:6789/0 conn(0x7f2b200d0720 legacy=0x7f2b200ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b2532e640 1 -- 172.21.15.1:0/2568675188 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b200cd7e0 con 0x7f2b200cd220 2024-03-24T10:34:20.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b24b2d640 1 -- 172.21.15.1:0/2568675188 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003589227 0 0) 0x7f2b200d0b50 con 0x7f2b200d0720 2024-03-24T10:34:20.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b2532e640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200cd220 0x7f2b200d01e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009870 tx=0x7f2b1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73a787902901ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.768+0000 7f2b24b2d640 1 -- 172.21.15.1:0/2568675188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14030840 con 0x7f2b200cd220 2024-03-24T10:34:20.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b24b2d640 1 -- 172.21.15.1:0/2568675188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b140309a0 con 0x7f2b200cd220 2024-03-24T10:34:20.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b24b2d640 1 -- 172.21.15.1:0/2568675188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14030d30 con 0x7f2b200cd220 2024-03-24T10:34:20.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 -- 172.21.15.1:0/2568675188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200cd220 msgr2=0x7f2b200d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200cd220 0x7f2b200d01e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009870 tx=0x7f2b1402b980 comp rx=0 tx=0).stop 2024-03-24T10:34:20.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 -- 172.21.15.1:0/2568675188 shutdown_connections 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b200cd220 0x7f2b200d01e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 --2- 172.21.15.1:0/2568675188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b2015bfa0 0x7f2b200d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 -- 172.21.15.1:0/2568675188 >> 172.21.15.1:0/2568675188 conn(0x7f2b200d1c00 msgr2=0x7f2b200d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 -- 172.21.15.1:0/2568675188 shutdown_connections 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.769+0000 7f2b27d93640 1 -- 172.21.15.1:0/2568675188 wait complete. 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.770+0000 7f2b27d93640 1 Processor -- start 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.770+0000 7f2b27d93640 1 -- start start 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200cd220 0x7f2b2015e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200d0720 0x7f2b200c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2b200c7a70 con 0x7f2b2015bfa0 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f2b200c6e60 con 0x7f2b200cd220 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b27d93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2b200c7140 con 0x7f2b200d0720 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b26330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b26330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34786/0 (socket says 172.21.15.1:34786) 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b26330640 1 -- 172.21.15.1:0/1667116435 learned_addr learned my addr 172.21.15.1:0/1667116435 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b2532e640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200d0720 0x7f2b200c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.771+0000 7f2b25b2f640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200cd220 0x7f2b2015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 -- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200d0720 msgr2=0x7f2b200c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200d0720 0x7f2b200c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 -- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200cd220 msgr2=0x7f2b2015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200cd220 0x7f2b2015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 -- 172.21.15.1:0/1667116435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b140094d0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b26330640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c7a0 tx=0x7f2b1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000a640 con 0x7f2b2015bfa0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b1000a7a0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b200c7420 con 0x7f2b2015bfa0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000abd0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.772+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b200cb4c0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.773+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b200c7be0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.773+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2b1000ad30 con 0x7f2b2015bfa0 2024-03-24T10:34:20.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.774+0000 7f2b1e7fc640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 0x7f2b0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:20.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.774+0000 7f2b25b2f640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 0x7f2b0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:20.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.774+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b100899a0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.774+0000 7f2b25b2f640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 0x7f2b0c06dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009aa0 tx=0x7f2b18009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:20.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.778+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2b10054470 con 0x7f2b2015bfa0 2024-03-24T10:34:20.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.960+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2b2004f470 con 0x7f2b2015bfa0 2024-03-24T10:34:20.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.960+0000 7f2b1e7fc640 1 -- 172.21.15.1:0/1667116435 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f2b100180a0 con 0x7f2b2015bfa0 2024-03-24T10:34:20.961 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.962+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 msgr2=0x7f2b0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 0x7f2b0c06dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009aa0 tx=0x7f2b18009190 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 msgr2=0x7f2b201f0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c7a0 tx=0x7f2b1000cc60 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 shutdown_connections 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b0c06b810 0x7f2b0c06dcd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b2015bfa0 0x7f2b201f0e90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2b200d0720 0x7f2b200c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 --2- 172.21.15.1:0/1667116435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b200cd220 0x7f2b2015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:20.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 >> 172.21.15.1:0/1667116435 conn(0x7f2b200d1c00 msgr2=0x7f2b200b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:20.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 shutdown_connections 2024-03-24T10:34:20.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:20.963+0000 7f2b27d93640 1 -- 172.21.15.1:0/1667116435 wait complete. 2024-03-24T10:34:20.974 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:21.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.078+0000 7f106ef5e640 1 Processor -- start 2024-03-24T10:34:21.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.078+0000 7f106ef5e640 1 -- start start 2024-03-24T10:34:21.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ef5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1068145730 0x7f106815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ef5e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106815ff10 0x7f10680cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ef5e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1068160340 con 0x7f106806d120 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ef5e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1068145b10 con 0x7f1068145730 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ef5e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f106806d620 con 0x7f106815ff10 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1068145730 0x7f106815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1068145730 0x7f106815f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34790/0 (socket says 172.21.15.1:34790) 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 -- 172.21.15.1:0/341815789 learned_addr learned my addr 172.21.15.1:0/341815789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106d4fb640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106815ff10 0x7f10680cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 -- 172.21.15.1:0/341815789 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106815ff10 msgr2=0x7f10680cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106815ff10 0x7f10680cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 -- 172.21.15.1:0/341815789 >> v1:172.21.15.3:6789/0 conn(0x7f106806d120 legacy=0x7f1068070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.079+0000 7f106ccfa640 1 -- 172.21.15.1:0/341815789 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106806d900 con 0x7f1068145730 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.080+0000 7f106ccfa640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815f9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1060009e50 tx=0x7f106002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a2b466b95a3830 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.080+0000 7f10677fe640 1 -- 172.21.15.1:0/341815789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106002cb00 con 0x7f1068145730 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.080+0000 7f10677fe640 1 -- 172.21.15.1:0/341815789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f106002cc60 con 0x7f1068145730 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.080+0000 7f10677fe640 1 -- 172.21.15.1:0/341815789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10600343f0 con 0x7f1068145730 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 -- 172.21.15.1:0/341815789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 msgr2=0x7f106815f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815f9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1060009e50 tx=0x7f106002bf60 comp rx=0 tx=0).stop 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 -- 172.21.15.1:0/341815789 shutdown_connections 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106815ff10 0x7f10680cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 --2- 172.21.15.1:0/341815789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815f9d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 -- 172.21.15.1:0/341815789 >> 172.21.15.1:0/341815789 conn(0x7f10680b1710 msgr2=0x7f10680b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 -- 172.21.15.1:0/341815789 shutdown_connections 2024-03-24T10:34:21.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.081+0000 7f106ef5e640 1 -- 172.21.15.1:0/341815789 wait complete. 2024-03-24T10:34:21.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.082+0000 7f106ef5e640 1 Processor -- start 2024-03-24T10:34:21.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.082+0000 7f106ef5e640 1 -- start start 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106806d120 0x7f10681f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106815ff10 0x7f10680b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10680b34e0 con 0x7f1068145730 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f10680b43b0 con 0x7f106815ff10 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ef5e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f10680b2bb0 con 0x7f106806d120 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34794/0 (socket says 172.21.15.1:34794) 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 -- 172.21.15.1:0/3485557469 learned_addr learned my addr 172.21.15.1:0/3485557469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106ccfa640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106806d120 0x7f10681f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f106d4fb640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106815ff10 0x7f10680b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 -- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106806d120 msgr2=0x7f10681f40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106806d120 0x7f10681f40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 -- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106815ff10 msgr2=0x7f10680b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106815ff10 0x7f10680b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.083+0000 7f1067fff640 1 -- 172.21.15.1:0/3485557469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1060009b30 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f1067fff640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f105c00c3c0 tx=0x7f105c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c0052d0 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c005450 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10680b2e90 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c003850 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.084+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10681f45f0 con 0x7f1068145730 2024-03-24T10:34:21.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.085+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10680cf340 con 0x7f1068145730 2024-03-24T10:34:21.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.085+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f105c0039b0 con 0x7f1068145730 2024-03-24T10:34:21.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.086+0000 7f10657fa640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 0x7f105406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.086+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f105c089650 con 0x7f1068145730 2024-03-24T10:34:21.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.086+0000 7f106ccfa640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 0x7f105406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.086+0000 7f106ccfa640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 0x7f105406dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1060035000 tx=0x7f1060030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.090+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f105c054120 con 0x7f1068145730 2024-03-24T10:34:21.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.271+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f106804f4e0 con 0x7f1068145730 2024-03-24T10:34:21.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.272+0000 7f10657fa640 1 -- 172.21.15.1:0/3485557469 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f105c0577e0 con 0x7f1068145730 2024-03-24T10:34:21.272 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 msgr2=0x7f105406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 0x7f105406dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1060035000 tx=0x7f1060030000 comp rx=0 tx=0).stop 2024-03-24T10:34:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 msgr2=0x7f106815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f105c00c3c0 tx=0x7f105c00c880 comp rx=0 tx=0).stop 2024-03-24T10:34:21.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 shutdown_connections 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f105406b810 0x7f105406dcd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106815ff10 0x7f10680b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1068145730 0x7f106815e710 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 --2- 172.21.15.1:0/3485557469 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f106806d120 0x7f10681f40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 >> 172.21.15.1:0/3485557469 conn(0x7f10680b1710 msgr2=0x7f10680ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 shutdown_connections 2024-03-24T10:34:21.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.274+0000 7f106ef5e640 1 -- 172.21.15.1:0/3485557469 wait complete. 2024-03-24T10:34:21.285 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:21.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.386+0000 7f147ca72640 1 Processor -- start 2024-03-24T10:34:21.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.386+0000 7f147ca72640 1 -- start start 2024-03-24T10:34:21.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478144770 0x7f14780d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1478070b90 0x7f147806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f14780b2aa0 0x7f14780b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f147806eaa0 con 0x7f1478070b90 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f14780b3470 con 0x7f14780b2aa0 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147ca72640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1478071010 con 0x7f1478144770 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1478070b90 0x7f147806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1478070b90 0x7f147806da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34796/0 (socket says 172.21.15.1:34796) 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 -- 172.21.15.1:0/3111248853 learned_addr learned my addr 172.21.15.1:0/3111248853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1476d9d640 1 --2- 172.21.15.1:0/3111248853 >> v2:172.21.15.3:3300/0 conn(0x7f14780b2aa0 0x7f14780b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f147659c640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478144770 0x7f14780d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 -- 172.21.15.1:0/3111248853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478144770 msgr2=0x7f14780d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478144770 0x7f14780d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 -- 172.21.15.1:0/3111248853 >> v2:172.21.15.3:3300/0 conn(0x7f14780b2aa0 msgr2=0x7f14780b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:21.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 --2- 172.21.15.1:0/3111248853 >> v2:172.21.15.3:3300/0 conn(0x7f14780b2aa0 0x7f14780b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.387+0000 7f1475d9b640 1 -- 172.21.15.1:0/3111248853 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14780cdd60 con 0x7f1478070b90 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.388+0000 7f1475d9b640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478070b90 0x7f147806da20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f14780af6e0 tx=0x7f146002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bc5b528d2e54b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.388+0000 7f147559a640 1 -- 172.21.15.1:0/3111248853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460031870 con 0x7f1478070b90 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.388+0000 7f147559a640 1 -- 172.21.15.1:0/3111248853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14600319d0 con 0x7f1478070b90 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.388+0000 7f147559a640 1 -- 172.21.15.1:0/3111248853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460031d60 con 0x7f1478070b90 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 -- 172.21.15.1:0/3111248853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478070b90 msgr2=0x7f147806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478070b90 0x7f147806da20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f14780af6e0 tx=0x7f146002b910 comp rx=0 tx=0).stop 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 -- 172.21.15.1:0/3111248853 shutdown_connections 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 --2- 172.21.15.1:0/3111248853 >> v2:172.21.15.3:3300/0 conn(0x7f14780b2aa0 0x7f14780b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478070b90 0x7f147806da20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 --2- 172.21.15.1:0/3111248853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478144770 0x7f14780d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 -- 172.21.15.1:0/3111248853 >> 172.21.15.1:0/3111248853 conn(0x7f14780b17a0 msgr2=0x7f14780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 -- 172.21.15.1:0/3111248853 shutdown_connections 2024-03-24T10:34:21.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.389+0000 7f147ca72640 1 -- 172.21.15.1:0/3111248853 wait complete. 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.390+0000 7f147ca72640 1 Processor -- start 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.390+0000 7f147ca72640 1 -- start start 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 0x7f147815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 0x7f14780aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 0x7f14781f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f147815dcd0 con 0x7f1478144770 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f14780aa690 con 0x7f14780b2aa0 2024-03-24T10:34:21.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147ca72640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f147815d050 con 0x7f1478070b90 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f1476d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 0x7f14781f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f1475d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 0x7f14780aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 0x7f147815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f1475d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 0x7f14780aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59842/0 (socket says 172.21.15.1:59842) 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f1475d9b640 1 -- 172.21.15.1:0/905203153 learned_addr learned my addr 172.21.15.1:0/905203153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f1476d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 0x7f14781f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34812/0 (socket says 172.21.15.1:34812) 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 -- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 msgr2=0x7f14780aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 0x7f14780aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 -- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 msgr2=0x7f14781f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 0x7f14781f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.391+0000 7f147659c640 1 -- 172.21.15.1:0/905203153 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14600094d0 con 0x7f1478070b90 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f147659c640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 0x7f147815c970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f146800ebc0 tx=0x7f146800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146800cda0 con 0x7f1478070b90 2024-03-24T10:34:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146800f040 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147815d330 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680146c0 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.392+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f147815d8f0 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.393+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f147804f4c0 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.393+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1468014820 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.394+0000 7f146effd640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 0x7f145806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.394+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f146808ad00 con 0x7f1478070b90 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.394+0000 7f1475d9b640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 0x7f145806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.394+0000 7f1475d9b640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 0x7f145806dd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f147815e320 tx=0x7f146003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.398+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f14680557e0 con 0x7f1478070b90 2024-03-24T10:34:21.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.579+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1478143b40 con 0x7f1478070b90 2024-03-24T10:34:21.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.580+0000 7f146effd640 1 -- 172.21.15.1:0/905203153 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f1468058ea0 con 0x7f1478070b90 2024-03-24T10:34:21.580 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:21.580 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 msgr2=0x7f145806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 0x7f145806dd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f147815e320 tx=0x7f146003c040 comp rx=0 tx=0).stop 2024-03-24T10:34:21.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 msgr2=0x7f147815c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 0x7f147815c970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f146800ebc0 tx=0x7f146800c490 comp rx=0 tx=0).stop 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 shutdown_connections 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f145806b840 0x7f145806dd00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1478144770 0x7f14781f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14780b2aa0 0x7f14780aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 --2- 172.21.15.1:0/905203153 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1478070b90 0x7f147815c970 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 >> 172.21.15.1:0/905203153 conn(0x7f14780b17a0 msgr2=0x7f1478143430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 shutdown_connections 2024-03-24T10:34:21.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.582+0000 7f147ca72640 1 -- 172.21.15.1:0/905203153 wait complete. 2024-03-24T10:34:21.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:21.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.699+0000 7efc24d02640 1 Processor -- start 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.699+0000 7efc24d02640 1 -- start start 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc24d02640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc2015c0e0 0x7efc200cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc24d02640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc200cfd40 0x7efc200d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc24d02640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc2015c4c0 con 0x7efc200ce590 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc24d02640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efc200cd5a0 con 0x7efc200cfd40 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc24d02640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efc200cd880 con 0x7efc2015c0e0 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc200cfd40 0x7efc200d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc200cfd40 0x7efc200d01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34818/0 (socket says 172.21.15.1:34818) 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 -- 172.21.15.1:0/2089719152 learned_addr learned my addr 172.21.15.1:0/2089719152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc1e59c640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc2015c0e0 0x7efc200cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 -- 172.21.15.1:0/2089719152 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc2015c0e0 msgr2=0x7efc200cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc2015c0e0 0x7efc200cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 -- 172.21.15.1:0/2089719152 >> v1:172.21.15.3:6789/0 conn(0x7efc200ce590 legacy=0x7efc200cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc17fff640 1 -- 172.21.15.1:0/2089719152 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc200cdb60 con 0x7efc200cfd40 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.700+0000 7efc1dd9b640 1 -- 172.21.15.1:0/2089719152 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3161933660 0 0) 0x7efc2015c4c0 con 0x7efc200ce590 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.701+0000 7efc17fff640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc200cfd40 0x7efc200d01a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d9bda624ffd175 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.701+0000 7efc1dd9b640 1 -- 172.21.15.1:0/2089719152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08030840 con 0x7efc200cfd40 2024-03-24T10:34:21.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.701+0000 7efc1dd9b640 1 -- 172.21.15.1:0/2089719152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc080309a0 con 0x7efc200cfd40 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.701+0000 7efc1dd9b640 1 -- 172.21.15.1:0/2089719152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08030d30 con 0x7efc200cfd40 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 -- 172.21.15.1:0/2089719152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc200cfd40 msgr2=0x7efc200d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc200cfd40 0x7efc200d01a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0802b980 comp rx=0 tx=0).stop 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 -- 172.21.15.1:0/2089719152 shutdown_connections 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc200cfd40 0x7efc200d01a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 --2- 172.21.15.1:0/2089719152 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc2015c0e0 0x7efc200cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 -- 172.21.15.1:0/2089719152 >> 172.21.15.1:0/2089719152 conn(0x7efc200d1c30 msgr2=0x7efc200d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 -- 172.21.15.1:0/2089719152 shutdown_connections 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.702+0000 7efc24d02640 1 -- 172.21.15.1:0/2089719152 wait complete. 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.703+0000 7efc24d02640 1 Processor -- start 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.703+0000 7efc24d02640 1 -- start start 2024-03-24T10:34:21.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 0x7efc200ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc200cfd40 0x7efc20150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc2015e9e0 con 0x7efc2015c0e0 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efc200ca5b0 con 0x7efc200cfd40 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc24d02640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efc200c9930 con 0x7efc200ce590 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc1ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc17fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc200cfd40 0x7efc20150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc1ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34826/0 (socket says 172.21.15.1:34826) 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc1e59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 0x7efc200ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc1ed9d640 1 -- 172.21.15.1:0/1787636419 learned_addr learned my addr 172.21.15.1:0/1787636419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.704+0000 7efc1e59c640 1 -- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc200cfd40 msgr2=0x7efc20150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1e59c640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc200cfd40 0x7efc20150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1e59c640 1 -- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 msgr2=0x7efc2006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1e59c640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1e59c640 1 -- 172.21.15.1:0/1787636419 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080094d0 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1ed9d640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc1e59c640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 0x7efc200ca9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efc1000e880 tx=0x7efc1000ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1000c940 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc1000caa0 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1000ce50 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc200c9c10 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.705+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc200ca1d0 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.706+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc200ca820 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.706+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efc10013470 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.707+0000 7efc16ffd640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 0x7efc0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.707+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efc1008a280 con 0x7efc200ce590 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.707+0000 7efc17fff640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 0x7efc0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:21.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.707+0000 7efc17fff640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 0x7efc0006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc08036000 tx=0x7efc08008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:21.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.711+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc10054d40 con 0x7efc200ce590 2024-03-24T10:34:21.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.892+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efc2004f4e0 con 0x7efc200ce590 2024-03-24T10:34:21.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.893+0000 7efc16ffd640 1 -- 172.21.15.1:0/1787636419 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7efc10058400 con 0x7efc200ce590 2024-03-24T10:34:21.893 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 msgr2=0x7efc0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 0x7efc0006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc08036000 tx=0x7efc08008030 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 msgr2=0x7efc200ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 0x7efc200ca9d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efc1000e880 tx=0x7efc1000ed40 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 shutdown_connections 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc0006b810 0x7efc0006dcd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc2015c0e0 0x7efc2006ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc200cfd40 0x7efc20150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 --2- 172.21.15.1:0/1787636419 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc200ce590 0x7efc200ca9d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 >> 172.21.15.1:0/1787636419 conn(0x7efc200d1c30 msgr2=0x7efc200d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 shutdown_connections 2024-03-24T10:34:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:21.895+0000 7efc24d02640 1 -- 172.21.15.1:0/1787636419 wait complete. 2024-03-24T10:34:21.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.010+0000 7f33d4556640 1 Processor -- start 2024-03-24T10:34:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.010+0000 7f33d4556640 1 -- start start 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.010+0000 7f33d4556640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 0x7f33cc168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d4556640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0cc250 con 0x7f33cc0b1380 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d4556640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0d1060 con 0x7f33cc0b31d0 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d4556640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0b35b0 con 0x7f33cc06d240 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d1af1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33cc0b1380 0x7f33cc06cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32964/0 (socket says 172.21.15.1:32964) 2024-03-24T10:34:22.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d1af1640 1 -- 172.21.15.1:0/3336474225 learned_addr learned my addr 172.21.15.1:0/3336474225 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d2af3640 1 --2- 172.21.15.1:0/3336474225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 0x7f33cc168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585690445 0 0) 0x7f33cc0cc250 con 0x7f33cc0b1380 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b8003520 con 0x7f33cc0b1380 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 839719188 0 0) 0x7f33cc0d1060 con 0x7f33cc0b31d0 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33cc0cc250 con 0x7f33cc0b31d0 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d2af3640 1 -- 172.21.15.1:0/3336474225 >> v1:172.21.15.3:6789/0 conn(0x7f33cc0b31d0 legacy=0x7f33cc0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d2af3640 1 -- 172.21.15.1:0/3336474225 >> v1:172.21.15.1:6789/0 conn(0x7f33cc0b1380 legacy=0x7f33cc06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.011+0000 7f33d2af3640 1 -- 172.21.15.1:0/3336474225 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0cc4a0 con 0x7f33cc06d240 2024-03-24T10:34:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.012+0000 7f33d2af3640 1 --2- 172.21.15.1:0/3336474225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 0x7f33cc168ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009dc0 tx=0x7f33c402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c3185e55dfcaf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.012+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c402ccd0 con 0x7f33cc06d240 2024-03-24T10:34:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.012+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c402ce30 con 0x7f33cc06d240 2024-03-24T10:34:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.012+0000 7f33d12f0640 1 -- 172.21.15.1:0/3336474225 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4034640 con 0x7f33cc06d240 2024-03-24T10:34:22.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 -- 172.21.15.1:0/3336474225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 msgr2=0x7f33cc168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 --2- 172.21.15.1:0/3336474225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 0x7f33cc168ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009dc0 tx=0x7f33c402bf20 comp rx=0 tx=0).stop 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 -- 172.21.15.1:0/3336474225 shutdown_connections 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 --2- 172.21.15.1:0/3336474225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc06d240 0x7f33cc168ea0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 -- 172.21.15.1:0/3336474225 >> 172.21.15.1:0/3336474225 conn(0x7f33cc0d1c00 msgr2=0x7f33cc0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 -- 172.21.15.1:0/3336474225 shutdown_connections 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.013+0000 7f33d4556640 1 -- 172.21.15.1:0/3336474225 wait complete. 2024-03-24T10:34:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 Processor -- start 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 -- start start 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33cc0b1380 0x7f33cc15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc0b31d0 0x7f33cc0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0c9b50 con 0x7f33cc06d240 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc15b2c0 con 0x7f33cc0b1380 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.014+0000 7f33d4556640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0c68e0 con 0x7f33cc0b31d0 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34836/0 (socket says 172.21.15.1:34836) 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 -- 172.21.15.1:0/3193977203 learned_addr learned my addr 172.21.15.1:0/3193977203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d1af1640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33cc0b1380 0x7f33cc15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d2af3640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc0b31d0 0x7f33cc0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 -- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc0b31d0 msgr2=0x7f33cc0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc0b31d0 0x7f33cc0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 -- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33cc0b1380 msgr2=0x7f33cc15ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33cc0b1380 0x7f33cc15ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 -- 172.21.15.1:0/3193977203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c4009a70 con 0x7f33cc06d240 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d22f2640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c430 tx=0x7f33c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80052d0 con 0x7f33cc06d240 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c8005450 con 0x7f33cc06d240 2024-03-24T10:34:22.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8003830 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc0c6bc0 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.015+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc0c7180 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.016+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc04f470 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.017+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f33c8003990 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.017+0000 7f33c2ffd640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 0x7f33b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.017+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f33c8089650 con 0x7f33cc06d240 2024-03-24T10:34:22.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.017+0000 7f33d1af1640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 0x7f33b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.018+0000 7f33d1af1640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 0x7f33b006dd00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0011e0 tx=0x7f33bc007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.021+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f33c80541b0 con 0x7f33cc06d240 2024-03-24T10:34:22.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.201+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33cc0b0af0 con 0x7f33cc06d240 2024-03-24T10:34:22.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.201+0000 7f33c2ffd640 1 -- 172.21.15.1:0/3193977203 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f33c8057870 con 0x7f33cc06d240 2024-03-24T10:34:22.202 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 msgr2=0x7f33b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 0x7f33b006dd00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0011e0 tx=0x7f33bc007d30 comp rx=0 tx=0).stop 2024-03-24T10:34:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 msgr2=0x7f33cc0c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c430 tx=0x7f33c800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 shutdown_connections 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f33b006b840 0x7f33b006dd00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f33cc0b31d0 0x7f33cc0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33cc0b1380 0x7f33cc15ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 --2- 172.21.15.1:0/3193977203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33cc06d240 0x7f33cc0c9610 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 >> 172.21.15.1:0/3193977203 conn(0x7f33cc0d1c00 msgr2=0x7f33cc0ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 shutdown_connections 2024-03-24T10:34:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.203+0000 7f33d4556640 1 -- 172.21.15.1:0/3193977203 wait complete. 2024-03-24T10:34:22.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.321+0000 7f54e8cc0640 1 Processor -- start 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.321+0000 7f54e8cc0640 1 -- start start 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e8cc0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 0x7f54e40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e8cc0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e8cc0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54e40b0b70 con 0x7f54e415e390 2024-03-24T10:34:22.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e8cc0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f54e40d1d60 con 0x7f54e40d2a20 2024-03-24T10:34:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e8cc0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f54e40b0de0 con 0x7f54e40b2310 2024-03-24T10:34:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f54e415e390 0x7f54e40d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32978/0 (socket says 172.21.15.1:32978) 2024-03-24T10:34:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e259c640 1 -- 172.21.15.1:0/2867037263 learned_addr learned my addr 172.21.15.1:0/2867037263 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e2d9d640 1 --2- 172.21.15.1:0/2867037263 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e1d9b640 1 --2- 172.21.15.1:0/2867037263 >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 0x7f54e40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e159a640 1 -- 172.21.15.1:0/2867037263 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807260001 0 0) 0x7f54e40b0b70 con 0x7f54e415e390 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e159a640 1 -- 172.21.15.1:0/2867037263 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54cc003520 con 0x7f54e415e390 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e2d9d640 1 -- 172.21.15.1:0/2867037263 >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 msgr2=0x7f54e40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e2d9d640 1 --2- 172.21.15.1:0/2867037263 >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 0x7f54e40cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e2d9d640 1 -- 172.21.15.1:0/2867037263 >> v1:172.21.15.1:6789/0 conn(0x7f54e415e390 legacy=0x7f54e40d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e2d9d640 1 -- 172.21.15.1:0/2867037263 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e415e770 con 0x7f54e40b2310 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.322+0000 7f54e1d9b640 1 --2- 172.21.15.1:0/2867037263 >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 0x7f54e40cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.323+0000 7f54e2d9d640 1 --2- 172.21.15.1:0/2867037263 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cf010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009d90 tx=0x7f54dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec1dba83f70d3a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.323+0000 7f54e159a640 1 -- 172.21.15.1:0/2867037263 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc02cc50 con 0x7f54e40b2310 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.323+0000 7f54e159a640 1 -- 172.21.15.1:0/2867037263 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54dc02cdb0 con 0x7f54e40b2310 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.323+0000 7f54e159a640 1 -- 172.21.15.1:0/2867037263 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc034640 con 0x7f54e40b2310 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 -- 172.21.15.1:0/2867037263 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 msgr2=0x7f54e40cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/2867037263 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cf010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009d90 tx=0x7f54dc02bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 -- 172.21.15.1:0/2867037263 shutdown_connections 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/2867037263 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cf010 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/2867037263 >> v2:172.21.15.3:3300/0 conn(0x7f54e40d2a20 0x7f54e40cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 -- 172.21.15.1:0/2867037263 >> 172.21.15.1:0/2867037263 conn(0x7f54e40d3ee0 msgr2=0x7f54e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 -- 172.21.15.1:0/2867037263 shutdown_connections 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.324+0000 7f54e8cc0640 1 -- 172.21.15.1:0/2867037263 wait complete. 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.325+0000 7f54e8cc0640 1 Processor -- start 2024-03-24T10:34:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.325+0000 7f54e8cc0640 1 -- start start 2024-03-24T10:34:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e415e390 0x7f54e415fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54e40ca8b0 con 0x7f54e40d2a20 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f54e40ca0a0 con 0x7f54e415e390 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e8cc0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f54e41f1510 con 0x7f54e40b2310 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34846/0 (socket says 172.21.15.1:34846) 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 -- 172.21.15.1:0/3948330092 learned_addr learned my addr 172.21.15.1:0/3948330092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e259c640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e2d9d640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e415e390 0x7f54e415fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 -- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 msgr2=0x7f54e40cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 -- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e415e390 msgr2=0x7f54e415fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e415e390 0x7f54e415fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 -- 172.21.15.1:0/3948330092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d8009400 con 0x7f54e40d2a20 2024-03-24T10:34:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.326+0000 7f54e1d9b640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54d800ebe0 tx=0x7f54d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.327+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8009760 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.327+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d800cc30 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.327+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc009a70 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.327+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d800eee0 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.327+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e40ca2f0 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.328+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e404f4e0 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.328+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f54d8013470 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.328+0000 7f54d2ffd640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 0x7f54c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.329+0000 7f54e259c640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 0x7f54c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.329+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f54d808a3e0 con 0x7f54e40d2a20 2024-03-24T10:34:22.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.329+0000 7f54e259c640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 0x7f54c406dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54d4005880 tx=0x7f54d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.333+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f54d8054eb0 con 0x7f54e40d2a20 2024-03-24T10:34:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.512+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f54e415f380 con 0x7f54e40d2a20 2024-03-24T10:34:22.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.513+0000 7f54d2ffd640 1 -- 172.21.15.1:0/3948330092 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f54d8058570 con 0x7f54e40d2a20 2024-03-24T10:34:22.513 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 msgr2=0x7f54c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 0x7f54c406dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54d4005880 tx=0x7f54d4009040 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 msgr2=0x7f54e40c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54d800ebe0 tx=0x7f54d800c520 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 shutdown_connections 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54c406b810 0x7f54c406dcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e415e390 0x7f54e415fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e40d2a20 0x7f54e40c7990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 --2- 172.21.15.1:0/3948330092 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e40b2310 0x7f54e40cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.514+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 >> 172.21.15.1:0/3948330092 conn(0x7f54e40d3ee0 msgr2=0x7f54e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.515+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 shutdown_connections 2024-03-24T10:34:22.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.515+0000 7f54e8cc0640 1 -- 172.21.15.1:0/3948330092 wait complete. 2024-03-24T10:34:22.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 Processor -- start 2024-03-24T10:34:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 -- start start 2024-03-24T10:34:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9d940d1000 0x7f9d940b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d9415e770 con 0x7f9d940d2bb0 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d940d4e40 con 0x7f9d9415e390 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.628+0000 7f9d9a6c3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9d940b3190 con 0x7f9d940d1000 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d937fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9d940d2bb0 0x7f9d940b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32992/0 (socket says 172.21.15.1:32992) 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d937fe640 1 -- 172.21.15.1:0/2301274168 learned_addr learned my addr 172.21.15.1:0/2301274168 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d98c60640 1 --2- 172.21.15.1:0/2301274168 >> v2:172.21.15.3:3300/0 conn(0x7f9d940d1000 0x7f9d940b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477415308 0 0) 0x7f9d88001110 con 0x7f9d940d2bb0 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d7c003520 con 0x7f9d940d2bb0 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d98c60640 1 -- 172.21.15.1:0/2301274168 >> v1:172.21.15.3:6789/0 conn(0x7f9d9415e390 legacy=0x7f9d940d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d98c60640 1 -- 172.21.15.1:0/2301274168 >> v1:172.21.15.1:6789/0 conn(0x7f9d940d2bb0 legacy=0x7f9d940b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3143360878 0 0) 0x7f9d940d4e40 con 0x7f9d9415e390 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.629+0000 7f9d98c60640 1 -- 172.21.15.1:0/2301274168 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d940d1e40 con 0x7f9d940d1000 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d98c60640 1 --2- 172.21.15.1:0/2301274168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d940b42a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009d90 tx=0x7f9d8c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4bbb9b54e4e558b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c02cbd0 con 0x7f9d940d1000 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c02cd30 con 0x7f9d940d1000 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d92ffd640 1 -- 172.21.15.1:0/2301274168 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c02cea0 con 0x7f9d940d1000 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/2301274168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 msgr2=0x7f9d940b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/2301274168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d940b42a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009d90 tx=0x7f9d8c02be20 comp rx=0 tx=0).stop 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/2301274168 shutdown_connections 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/2301274168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d940b42a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/2301274168 >> 172.21.15.1:0/2301274168 conn(0x7f9d940d3ec0 msgr2=0x7f9d940d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/2301274168 shutdown_connections 2024-03-24T10:34:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.630+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/2301274168 wait complete. 2024-03-24T10:34:22.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 Processor -- start 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 -- start start 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d941480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9415e390 0x7f9d9415a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.632+0000 7f9d9a6c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9d941e8ac0 con 0x7f9d940d2bb0 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d9a6c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9d94148620 con 0x7f9d940d1000 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d9a6c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9d940cdd50 con 0x7f9d9415e390 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d93fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d941480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34854/0 (socket says 172.21.15.1:34854) 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 -- 172.21.15.1:0/1941177174 learned_addr learned my addr 172.21.15.1:0/1941177174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d98c60640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9415e390 0x7f9d9415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 -- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9415e390 msgr2=0x7f9d9415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9415e390 0x7f9d9415a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 -- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 msgr2=0x7f9d941480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d941480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 -- 172.21.15.1:0/1941177174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c009a70 con 0x7f9d940d2bb0 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d93fff640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d941480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d937fe640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800c700 tx=0x7f9d8800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88005ab0 con 0x7f9d940d2bb0 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d88005c10 con 0x7f9d940d2bb0 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d941e8500 con 0x7f9d940d2bb0 2024-03-24T10:34:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88001d50 con 0x7f9d940d2bb0 2024-03-24T10:34:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.633+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d941599e0 con 0x7f9d940d2bb0 2024-03-24T10:34:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.634+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d9415a790 con 0x7f9d940d2bb0 2024-03-24T10:34:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.635+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9d88007070 con 0x7f9d940d2bb0 2024-03-24T10:34:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.635+0000 7f9d90ff9640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 0x7f9d8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.635+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9d88089510 con 0x7f9d940d2bb0 2024-03-24T10:34:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.635+0000 7f9d93fff640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 0x7f9d8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.636+0000 7f9d93fff640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 0x7f9d8006dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9d84000f60 tx=0x7f9d84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.639+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9d9415a790 con 0x7f9d940d2bb0 2024-03-24T10:34:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.822+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d9415e770 con 0x7f9d940d2bb0 2024-03-24T10:34:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.823+0000 7f9d90ff9640 1 -- 172.21.15.1:0/1941177174 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f9d9415e770 con 0x7f9d940d2bb0 2024-03-24T10:34:22.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:22.823 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 msgr2=0x7f9d8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 0x7f9d8006dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9d84000f60 tx=0x7f9d84009040 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 msgr2=0x7f9d94159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800c700 tx=0x7f9d8800cbc0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 shutdown_connections 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8006b810 0x7f9d8006dcd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9415e390 0x7f9d9415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d940d2bb0 0x7f9d94159cf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 --2- 172.21.15.1:0/1941177174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d940d1000 0x7f9d941480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 >> 172.21.15.1:0/1941177174 conn(0x7f9d940d3ec0 msgr2=0x7f9d940b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 shutdown_connections 2024-03-24T10:34:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.825+0000 7f9d9a6c3640 1 -- 172.21.15.1:0/1941177174 wait complete. 2024-03-24T10:34:22.836 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:22.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.941+0000 7f9b771b5640 1 Processor -- start 2024-03-24T10:34:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.941+0000 7f9b771b5640 1 -- start start 2024-03-24T10:34:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b771b5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b7006e480 0x7f9b700cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b771b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b700cf1c0 con 0x7f9b7006f550 2024-03-24T10:34:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b771b5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b700ce8d0 con 0x7f9b7015bd00 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b771b5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7006ebf0 con 0x7f9b7006e480 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b7006e480 0x7f9b700cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b7006e480 0x7f9b700cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34864/0 (socket says 172.21.15.1:34864) 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b75752640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9b7006f550 0x7f9b7006b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33000/0 (socket says 172.21.15.1:33000) 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 -- 172.21.15.1:0/3608206152 learned_addr learned my addr 172.21.15.1:0/3608206152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 -- 172.21.15.1:0/3608206152 >> v1:172.21.15.3:6789/0 conn(0x7f9b7015bd00 legacy=0x7f9b700cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 -- 172.21.15.1:0/3608206152 >> v1:172.21.15.1:6789/0 conn(0x7f9b7006f550 legacy=0x7f9b7006b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6f7fe640 1 -- 172.21.15.1:0/3608206152 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764523454 0 0) 0x7f9b700ce8d0 con 0x7f9b7015bd00 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.942+0000 7f9b6ffff640 1 -- 172.21.15.1:0/3608206152 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7006e910 con 0x7f9b7006e480 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.943+0000 7f9b6ffff640 1 --2- 172.21.15.1:0/3608206152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006e480 0x7f9b700cc8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009870 tx=0x7f9b5c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc796fefdf7ebf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.943+0000 7f9b6f7fe640 1 -- 172.21.15.1:0/3608206152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c031740 con 0x7f9b7006e480 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.943+0000 7f9b6f7fe640 1 -- 172.21.15.1:0/3608206152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c0318a0 con 0x7f9b7006e480 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.943+0000 7f9b6f7fe640 1 -- 172.21.15.1:0/3608206152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c031c30 con 0x7f9b7006e480 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 -- 172.21.15.1:0/3608206152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006e480 msgr2=0x7f9b700cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 --2- 172.21.15.1:0/3608206152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006e480 0x7f9b700cc8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009870 tx=0x7f9b5c02b860 comp rx=0 tx=0).stop 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 -- 172.21.15.1:0/3608206152 shutdown_connections 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 --2- 172.21.15.1:0/3608206152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006e480 0x7f9b700cc8b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 -- 172.21.15.1:0/3608206152 >> 172.21.15.1:0/3608206152 conn(0x7f9b700d02e0 msgr2=0x7f9b700d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 -- 172.21.15.1:0/3608206152 shutdown_connections 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.944+0000 7f9b771b5640 1 -- 172.21.15.1:0/3608206152 wait complete. 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.945+0000 7f9b771b5640 1 Processor -- start 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.945+0000 7f9b771b5640 1 -- start start 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7015bd00 0x7f9b70156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b70148f90 0x7f9b7015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9b700cfd60 con 0x7f9b7006f550 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9b70152480 con 0x7f9b7015bd00 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b771b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9b701553b0 con 0x7f9b70148f90 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34868/0 (socket says 172.21.15.1:34868) 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 -- 172.21.15.1:0/1586611985 learned_addr learned my addr 172.21.15.1:0/1586611985 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b6ffff640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7015bd00 0x7f9b70156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b75752640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b70148f90 0x7f9b7015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 -- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b70148f90 msgr2=0x7f9b7015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b70148f90 0x7f9b7015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 -- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7015bd00 msgr2=0x7f9b70156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7015bd00 0x7f9b70156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 -- 172.21.15.1:0/1586611985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5c0094d0 con 0x7f9b7006f550 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b74f51640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400c440 tx=0x7f9b6400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64005480 con 0x7f9b7006f550 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.946+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b640035f0 con 0x7f9b7006f550 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.947+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b70156d40 con 0x7f9b7006f550 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.947+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b640039a0 con 0x7f9b7006f550 2024-03-24T10:34:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.947+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b70151e30 con 0x7f9b7006f550 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.948+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b7004f4e0 con 0x7f9b7006f550 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.948+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9b64003b00 con 0x7f9b7006f550 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.948+0000 7f9b6d7fa640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 0x7f9b5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.948+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9b64089660 con 0x7f9b7006f550 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.948+0000 7f9b6ffff640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 0x7f9b5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.949+0000 7f9b6ffff640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 0x7f9b5406dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c035000 tx=0x7f9b5c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:22.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:22.953+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9b64054130 con 0x7f9b7006f550 2024-03-24T10:34:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.136+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9b7015fe80 con 0x7f9b7006f550 2024-03-24T10:34:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.136+0000 7f9b6d7fa640 1 -- 172.21.15.1:0/1586611985 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f9b640577f0 con 0x7f9b7006f550 2024-03-24T10:34:23.136 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 msgr2=0x7f9b5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 0x7f9b5406dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c035000 tx=0x7f9b5c008030 comp rx=0 tx=0).stop 2024-03-24T10:34:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 msgr2=0x7f9b70154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400c440 tx=0x7f9b6400c900 comp rx=0 tx=0).stop 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 shutdown_connections 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9b5406b810 0x7f9b5406dcd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9b70148f90 0x7f9b7015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7015bd00 0x7f9b70156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 --2- 172.21.15.1:0/1586611985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b7006f550 0x7f9b70154e70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.138+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 >> 172.21.15.1:0/1586611985 conn(0x7f9b700d02e0 msgr2=0x7f9b7015ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.139+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 shutdown_connections 2024-03-24T10:34:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.139+0000 7f9b771b5640 1 -- 172.21.15.1:0/1586611985 wait complete. 2024-03-24T10:34:23.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:23.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.254+0000 7ff51f681640 1 Processor -- start 2024-03-24T10:34:23.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.254+0000 7ff51f681640 1 -- start start 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51f681640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff51815c1e0 0x7ff5180d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51f681640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff5180d2930 0x7ff5180cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51f681640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5180d0e30 con 0x7ff5180cd970 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51f681640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff5180cf8e0 con 0x7ff51815c1e0 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51f681640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7ff5180cfb30 con 0x7ff5180d2930 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51d41d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff51815c1e0 0x7ff5180d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51d41d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff51815c1e0 0x7ff5180d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34874/0 (socket says 172.21.15.1:34874) 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51d41d640 1 -- 172.21.15.1:0/3345013129 learned_addr learned my addr 172.21.15.1:0/3345013129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51dc1e640 1 --1- 172.21.15.1:0/3345013129 >> v1:172.21.15.1:6789/0 conn(0x7ff5180cd970 0x7ff5180ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33010/0 (socket says 172.21.15.1:33010) 2024-03-24T10:34:23.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.255+0000 7ff51cc1c640 1 --2- 172.21.15.1:0/3345013129 >> v2:172.21.15.3:3300/0 conn(0x7ff5180d2930 0x7ff5180cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff51d41d640 1 -- 172.21.15.1:0/3345013129 >> v2:172.21.15.3:3300/0 conn(0x7ff5180d2930 msgr2=0x7ff5180cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff51d41d640 1 --2- 172.21.15.1:0/3345013129 >> v2:172.21.15.3:3300/0 conn(0x7ff5180d2930 0x7ff5180cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff51d41d640 1 -- 172.21.15.1:0/3345013129 >> v1:172.21.15.1:6789/0 conn(0x7ff5180cd970 legacy=0x7ff5180ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff51d41d640 1 -- 172.21.15.1:0/3345013129 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5180cfe10 con 0x7ff51815c1e0 2024-03-24T10:34:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff51d41d640 1 --2- 172.21.15.1:0/3345013129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff51815c1e0 0x7ff5180d08f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff518071b40 tx=0x7ff51002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea4a5219c1b96b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.256+0000 7ff517fff640 1 -- 172.21.15.1:0/3345013129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51002f850 con 0x7ff51815c1e0 2024-03-24T10:34:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff517fff640 1 -- 172.21.15.1:0/3345013129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51002f9b0 con 0x7ff51815c1e0 2024-03-24T10:34:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff517fff640 1 -- 172.21.15.1:0/3345013129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51002fd40 con 0x7ff51815c1e0 2024-03-24T10:34:23.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 -- 172.21.15.1:0/3345013129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff51815c1e0 msgr2=0x7ff5180d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 --2- 172.21.15.1:0/3345013129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff51815c1e0 0x7ff5180d08f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff518071b40 tx=0x7ff51002ba30 comp rx=0 tx=0).stop 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 -- 172.21.15.1:0/3345013129 shutdown_connections 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 --2- 172.21.15.1:0/3345013129 >> v2:172.21.15.3:3300/0 conn(0x7ff5180d2930 0x7ff5180cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 --2- 172.21.15.1:0/3345013129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff51815c1e0 0x7ff5180d08f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 -- 172.21.15.1:0/3345013129 >> 172.21.15.1:0/3345013129 conn(0x7ff5180d1d10 msgr2=0x7ff5180d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 -- 172.21.15.1:0/3345013129 shutdown_connections 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.257+0000 7ff51f681640 1 -- 172.21.15.1:0/3345013129 wait complete. 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.258+0000 7ff51f681640 1 Processor -- start 2024-03-24T10:34:23.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.258+0000 7ff51f681640 1 -- start start 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5180cd970 0x7ff5181e64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff51815c1e0 0x7ff518159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181e6080 con 0x7ff5180d2930 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181e69e0 con 0x7ff5180cd970 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51f681640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff51806d050 con 0x7ff51815c1e0 2024-03-24T10:34:23.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34878/0 (socket says 172.21.15.1:34878) 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 -- 172.21.15.1:0/3903793013 learned_addr learned my addr 172.21.15.1:0/3903793013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51d41d640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5180cd970 0x7ff5181e64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51dc1e640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff51815c1e0 0x7ff518159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 -- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff51815c1e0 msgr2=0x7ff518159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff51815c1e0 0x7ff518159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 -- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5180cd970 msgr2=0x7ff5181e64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5180cd970 0x7ff5181e64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.259+0000 7ff51cc1c640 1 -- 172.21.15.1:0/3903793013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5100094d0 con 0x7ff5180d2930 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff51cc1c640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00e9b0 tx=0x7ff50c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c009670 con 0x7ff5180d2930 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff50c00c9a0 con 0x7ff5180d2930 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5181e56e0 con 0x7ff5180d2930 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c00cd50 con 0x7ff5180d2930 2024-03-24T10:34:23.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.260+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5181e5ca0 con 0x7ff5180d2930 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.261+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff51804f470 con 0x7ff5180d2930 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.261+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff50c013470 con 0x7ff5180d2930 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.262+0000 7ff515ffb640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 0x7ff50406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.262+0000 7ff51d41d640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 0x7ff50406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.262+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff50c08a370 con 0x7ff5180d2930 2024-03-24T10:34:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.262+0000 7ff51d41d640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 0x7ff50406dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff510036000 tx=0x7ff510008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.266+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff50c054e30 con 0x7ff5180d2930 2024-03-24T10:34:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.446+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff5180cc1d0 con 0x7ff5180d2930 2024-03-24T10:34:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.446+0000 7ff515ffb640 1 -- 172.21.15.1:0/3903793013 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7ff50c0584f0 con 0x7ff5180d2930 2024-03-24T10:34:23.446 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:23.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 msgr2=0x7ff50406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 0x7ff50406dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff510036000 tx=0x7ff510008030 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 msgr2=0x7ff518157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00e9b0 tx=0x7ff50c00ee70 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 shutdown_connections 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff50406b810 0x7ff50406dcd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff51815c1e0 0x7ff518159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5180d2930 0x7ff518157fe0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 --2- 172.21.15.1:0/3903793013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5180cd970 0x7ff5181e64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 >> 172.21.15.1:0/3903793013 conn(0x7ff5180d1d10 msgr2=0x7ff5180b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 shutdown_connections 2024-03-24T10:34:23.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.448+0000 7ff51f681640 1 -- 172.21.15.1:0/3903793013 wait complete. 2024-03-24T10:34:23.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.564+0000 7f6f3dd3a640 1 Processor -- start 2024-03-24T10:34:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.564+0000 7f6f3dd3a640 1 -- start start 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f3dd3a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f3815e2c0 0x7f6f3806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f3dd3a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f380cdd10 0x7f6f3806fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f3dd3a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f38070120 con 0x7f6f3806f780 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f3dd3a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6f380cc2b0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f3dd3a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6f380cc870 con 0x7f6f380cdd10 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f3815e2c0 0x7f6f3806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f3815e2c0 0x7f6f3806f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34894/0 (socket says 172.21.15.1:34894) 2024-03-24T10:34:23.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 -- 172.21.15.1:0/4143467503 learned_addr learned my addr 172.21.15.1:0/4143467503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f37fff640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f380cdd10 0x7f6f3806fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 -- 172.21.15.1:0/4143467503 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f380cdd10 msgr2=0x7f6f3806fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f380cdd10 0x7f6f3806fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 -- 172.21.15.1:0/4143467503 >> v1:172.21.15.3:6789/0 conn(0x7f6f3806f780 legacy=0x7f6f380ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f367fc640 1 -- 172.21.15.1:0/4143467503 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 37731383 0 0) 0x7f6f38070120 con 0x7f6f3806f780 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.565+0000 7f6f377fe640 1 -- 172.21.15.1:0/4143467503 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f380d0d50 con 0x7f6f3815e2c0 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.566+0000 7f6f377fe640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f3806f130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009870 tx=0x7f6f2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d5ae23ffe42a22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.566+0000 7f6f367fc640 1 -- 172.21.15.1:0/4143467503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28031640 con 0x7f6f3815e2c0 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.566+0000 7f6f367fc640 1 -- 172.21.15.1:0/4143467503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f280317a0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.566+0000 7f6f367fc640 1 -- 172.21.15.1:0/4143467503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28031b30 con 0x7f6f3815e2c0 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/4143467503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 msgr2=0x7f6f3806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f3806f130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009870 tx=0x7f6f2802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/4143467503 shutdown_connections 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f380cdd10 0x7f6f3806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/4143467503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f3806f130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/4143467503 >> 172.21.15.1:0/4143467503 conn(0x7f6f380d1c60 msgr2=0x7f6f380d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/4143467503 shutdown_connections 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.567+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/4143467503 wait complete. 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.568+0000 7f6f3dd3a640 1 Processor -- start 2024-03-24T10:34:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.568+0000 7f6f3dd3a640 1 -- start start 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f3806f780 0x7f6f38154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f380cdd10 0x7f6f381534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6f380b30d0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3815a230 con 0x7f6f380cdd10 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6f38154eb0 con 0x7f6f3806f780 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34898/0 (socket says 172.21.15.1:34898) 2024-03-24T10:34:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 -- 172.21.15.1:0/2785943983 learned_addr learned my addr 172.21.15.1:0/2785943983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f377fe640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f3806f780 0x7f6f38154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f36ffd640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f380cdd10 0x7f6f381534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 -- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f3806f780 msgr2=0x7f6f38154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f3806f780 0x7f6f38154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 -- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f380cdd10 msgr2=0x7f6f381534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f380cdd10 0x7f6f381534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 -- 172.21.15.1:0/2785943983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f280094d0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f37fff640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000efc0 tx=0x7f6f3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30009b20 con 0x7f6f3815e2c0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f3000f040 con 0x7f6f3815e2c0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f381550e0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.569+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f300146c0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.570+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f380b35a0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.570+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f3804f4e0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.571+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f30014820 con 0x7f6f3815e2c0 2024-03-24T10:34:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.571+0000 7f6f1ffff640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 0x7f6f2406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.571+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6f3008ae20 con 0x7f6f3815e2c0 2024-03-24T10:34:23.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.571+0000 7f6f377fe640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 0x7f6f2406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.572+0000 7f6f377fe640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 0x7f6f2406dd00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f28036000 tx=0x7f6f2803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.576+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f30055900 con 0x7f6f3815e2c0 2024-03-24T10:34:23.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.756+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f380c84e0 con 0x7f6f3815e2c0 2024-03-24T10:34:23.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.756+0000 7f6f1ffff640 1 -- 172.21.15.1:0/2785943983 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f6f30009e60 con 0x7f6f3815e2c0 2024-03-24T10:34:23.756 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:23.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 msgr2=0x7f6f2406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 0x7f6f2406dd00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f28036000 tx=0x7f6f2803d040 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 msgr2=0x7f6f38156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000efc0 tx=0x7f6f3000c490 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 shutdown_connections 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f2406b840 0x7f6f2406dd00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f3815e2c0 0x7f6f38156540 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f380cdd10 0x7f6f381534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 --2- 172.21.15.1:0/2785943983 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f3806f780 0x7f6f38154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 >> 172.21.15.1:0/2785943983 conn(0x7f6f380d1c60 msgr2=0x7f6f3806fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 shutdown_connections 2024-03-24T10:34:23.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.758+0000 7f6f3dd3a640 1 -- 172.21.15.1:0/2785943983 wait complete. 2024-03-24T10:34:23.795 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-24T10:34:23.795 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:34:23.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.897+0000 7f63f60ae640 1 Processor -- start 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.897+0000 7f63f60ae640 1 -- start start 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63f60ae640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63f006d010 0x7f63f00b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63f60ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63f00d10c0 con 0x7f63f006cba0 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63f60ae640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63f00b4c70 con 0x7f63f00b7530 2024-03-24T10:34:23.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63f60ae640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f63f00d1320 con 0x7f63f006d010 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63eeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f63f006cba0 0x7f63f00b5a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33020/0 (socket says 172.21.15.1:33020) 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63eeffd640 1 -- 172.21.15.1:0/3926108584 learned_addr learned my addr 172.21.15.1:0/3926108584 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63effff640 1 --2- 172.21.15.1:0/3926108584 >> v2:172.21.15.3:3300/0 conn(0x7f63f006d010 0x7f63f00b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.898+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909761942 0 0) 0x7f63f00d10c0 con 0x7f63f006cba0 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63d8003520 con 0x7f63f006cba0 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141175750 0 0) 0x7f63f00b4c70 con 0x7f63f00b7530 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63f00d10c0 con 0x7f63f00b7530 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63effff640 1 -- 172.21.15.1:0/3926108584 >> v1:172.21.15.3:6789/0 conn(0x7f63f00b7530 legacy=0x7f63f00d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63effff640 1 -- 172.21.15.1:0/3926108584 >> v1:172.21.15.1:6789/0 conn(0x7f63f006cba0 legacy=0x7f63f00b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63effff640 1 -- 172.21.15.1:0/3926108584 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f00b4f50 con 0x7f63f006d010 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.899+0000 7f63effff640 1 --2- 172.21.15.1:0/3926108584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f006d010 0x7f63f00b6140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009d90 tx=0x7f63e802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be59ad3b69945624 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e802cd00 con 0x7f63f006d010 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e802ce60 con 0x7f63f006d010 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63ee7fc640 1 -- 172.21.15.1:0/3926108584 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8034660 con 0x7f63f006d010 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 -- 172.21.15.1:0/3926108584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f006d010 msgr2=0x7f63f00b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 --2- 172.21.15.1:0/3926108584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f006d010 0x7f63f00b6140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009d90 tx=0x7f63e802bf50 comp rx=0 tx=0).stop 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 -- 172.21.15.1:0/3926108584 shutdown_connections 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 --2- 172.21.15.1:0/3926108584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f006d010 0x7f63f00b6140 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 -- 172.21.15.1:0/3926108584 >> 172.21.15.1:0/3926108584 conn(0x7f63f00d1c80 msgr2=0x7f63f00d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 -- 172.21.15.1:0/3926108584 shutdown_connections 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.900+0000 7f63f60ae640 1 -- 172.21.15.1:0/3926108584 wait complete. 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.901+0000 7f63f60ae640 1 Processor -- start 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.901+0000 7f63f60ae640 1 -- start start 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63f006d010 0x7f63f00c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f00b7530 0x7f63f014ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f63f015f070 con 0x7f63f006cba0 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f63f015f350 con 0x7f63f00b7530 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63f60ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63f00ca1b0 con 0x7f63f006d010 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63ef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63effff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f00b7530 0x7f63f014ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63ef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34906/0 (socket says 172.21.15.1:34906) 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63ef7fe640 1 -- 172.21.15.1:0/1210288256 learned_addr learned my addr 172.21.15.1:0/1210288256 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.902+0000 7f63eeffd640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63f006d010 0x7f63f00c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 -- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63f006d010 msgr2=0x7f63f00c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63f006d010 0x7f63f00c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 -- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f00b7530 msgr2=0x7f63f014ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f00b7530 0x7f63f014ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 -- 172.21.15.1:0/1210288256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e8009a70 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63ef7fe640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c430 tx=0x7f63e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e00052d0 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e0005450 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f00ca490 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0003830 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.903+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f015c9b0 con 0x7f63f006cba0 2024-03-24T10:34:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.904+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f015cba0 con 0x7f63f006cba0 2024-03-24T10:34:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.905+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f63e0003990 con 0x7f63f006cba0 2024-03-24T10:34:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.905+0000 7f63d7fff640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 0x7f63dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.905+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f63e0089760 con 0x7f63f006cba0 2024-03-24T10:34:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.905+0000 7f63eeffd640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 0x7f63dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.905+0000 7f63eeffd640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 0x7f63dc06dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63f006fa30 tx=0x7f63e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:23.909+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63e0054230 con 0x7f63f006cba0 2024-03-24T10:34:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.091+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63f006de20 con 0x7f63f006cba0 2024-03-24T10:34:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.092+0000 7f63d7fff640 1 -- 172.21.15.1:0/1210288256 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f63e00578f0 con 0x7f63f006cba0 2024-03-24T10:34:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:24.093 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 msgr2=0x7f63dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 0x7f63dc06dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f63f006fa30 tx=0x7f63e4009040 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 msgr2=0x7f63f015c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c430 tx=0x7f63e000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 shutdown_connections 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63dc06b810 0x7f63dc06dcd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63f00b7530 0x7f63f014ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63f006d010 0x7f63f00c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 --2- 172.21.15.1:0/1210288256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63f006cba0 0x7f63f015c470 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 >> 172.21.15.1:0/1210288256 conn(0x7f63f00d1c80 msgr2=0x7f63f006f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 shutdown_connections 2024-03-24T10:34:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.094+0000 7f63f60ae640 1 -- 172.21.15.1:0/1210288256 wait complete. 2024-03-24T10:34:24.106 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:24.106 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.210+0000 7fbb173c7640 1 Processor -- start 2024-03-24T10:34:24.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb173c7640 1 -- start start 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb173c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbb1006f6b0 0x7fbb100d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb173c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb100cf340 con 0x7fbb1006b0a0 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb173c7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb10145b10 con 0x7fbb1006e6c0 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb173c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1006ef40 con 0x7fbb1006f6b0 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.211+0000 7fbb15163640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb1006b0a0 0x7fbb1006b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33022/0 (socket says 172.21.15.1:33022) 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb15163640 1 -- 172.21.15.1:0/2794993698 learned_addr learned my addr 172.21.15.1:0/2794993698 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb14962640 1 --2- 172.21.15.1:0/2794993698 >> v2:172.21.15.1:3300/0 conn(0x7fbb1006f6b0 0x7fbb100d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579623600 0 0) 0x7fbb100cf340 con 0x7fbb1006b0a0 2024-03-24T10:34:24.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaf8003520 con 0x7fbb1006b0a0 2024-03-24T10:34:24.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb14962640 1 -- 172.21.15.1:0/2794993698 >> v1:172.21.15.3:6789/0 conn(0x7fbb1006e6c0 legacy=0x7fbb1015c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189512726 0 0) 0x7fbb08001110 con 0x7fbb1006e6c0 2024-03-24T10:34:24.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb14962640 1 -- 172.21.15.1:0/2794993698 >> v1:172.21.15.1:6789/0 conn(0x7fbb1006b0a0 legacy=0x7fbb1006b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.212+0000 7fbb14962640 1 -- 172.21.15.1:0/2794993698 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb100cf570 con 0x7fbb1006f6b0 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb14962640 1 --2- 172.21.15.1:0/2794993698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006f6b0 0x7fbb100d0330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb040097a0 tx=0x7fbb0402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630109e6265efab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04030610 con 0x7fbb1006f6b0 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb04030770 con 0x7fbb1006f6b0 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb0ffff640 1 -- 172.21.15.1:0/2794993698 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04030b00 con 0x7fbb1006f6b0 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 -- 172.21.15.1:0/2794993698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006f6b0 msgr2=0x7fbb100d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 --2- 172.21.15.1:0/2794993698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006f6b0 0x7fbb100d0330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb040097a0 tx=0x7fbb0402b960 comp rx=0 tx=0).stop 2024-03-24T10:34:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 -- 172.21.15.1:0/2794993698 shutdown_connections 2024-03-24T10:34:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 --2- 172.21.15.1:0/2794993698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006f6b0 0x7fbb100d0330 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 -- 172.21.15.1:0/2794993698 >> 172.21.15.1:0/2794993698 conn(0x7fbb100b2890 msgr2=0x7fbb100b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 -- 172.21.15.1:0/2794993698 shutdown_connections 2024-03-24T10:34:24.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.213+0000 7fbb173c7640 1 -- 172.21.15.1:0/2794993698 wait complete. 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.214+0000 7fbb173c7640 1 Processor -- start 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 -- start start 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb1006b0a0 0x7fbb10155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb1006f6b0 0x7fbb10158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101e56d0 con 0x7fbb1006e6c0 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101e59b0 con 0x7fbb1006f6b0 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb173c7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb101e5c90 con 0x7fbb1006b0a0 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb14962640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb14962640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34926/0 (socket says 172.21.15.1:34926) 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb14962640 1 -- 172.21.15.1:0/904391578 learned_addr learned my addr 172.21.15.1:0/904391578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb15163640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb1006b0a0 0x7fbb10155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.215+0000 7fbb15964640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb1006f6b0 0x7fbb10158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 -- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb1006b0a0 msgr2=0x7fbb10155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb1006b0a0 0x7fbb10155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 -- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb1006f6b0 msgr2=0x7fbb10158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb1006f6b0 0x7fbb10158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 -- 172.21.15.1:0/904391578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb04009400 con 0x7fbb1006e6c0 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb14962640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbb04000c00 tx=0x7fbb04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0403e040 con 0x7fbb1006e6c0 2024-03-24T10:34:24.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb04039c90 con 0x7fbb1006e6c0 2024-03-24T10:34:24.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04030e30 con 0x7fbb1006e6c0 2024-03-24T10:34:24.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb101e5f70 con 0x7fbb1006e6c0 2024-03-24T10:34:24.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.216+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb10156020 con 0x7fbb1006e6c0 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.217+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1004f4e0 con 0x7fbb1006e6c0 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.218+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb0404d3b0 con 0x7fbb1006e6c0 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.218+0000 7fbb0dffb640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 0x7fbafc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.218+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb040ac640 con 0x7fbb1006e6c0 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.218+0000 7fbb15163640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 0x7fbafc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.219+0000 7fbb15163640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 0x7fbafc06dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb00005c00 tx=0x7fbb00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.224+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb04077110 con 0x7fbb1006e6c0 2024-03-24T10:34:24.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.406+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbb101e5810 con 0x7fbb1006e6c0 2024-03-24T10:34:24.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.406+0000 7fbb0dffb640 1 -- 172.21.15.1:0/904391578 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fbb101e5810 con 0x7fbb1006e6c0 2024-03-24T10:34:24.406 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:24.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 msgr2=0x7fbafc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 0x7fbafc06dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb00005c00 tx=0x7fbb00009040 comp rx=0 tx=0).stop 2024-03-24T10:34:24.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 msgr2=0x7fbb10158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbb04000c00 tx=0x7fbb04004730 comp rx=0 tx=0).stop 2024-03-24T10:34:24.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 shutdown_connections 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbafc06b810 0x7fbafc06dcd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb1006f6b0 0x7fbb10158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb1006e6c0 0x7fbb10158500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 --2- 172.21.15.1:0/904391578 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb1006b0a0 0x7fbb10155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 >> 172.21.15.1:0/904391578 conn(0x7fbb100b2890 msgr2=0x7fbb1006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 shutdown_connections 2024-03-24T10:34:24.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.408+0000 7fbb173c7640 1 -- 172.21.15.1:0/904391578 wait complete. 2024-03-24T10:34:24.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.524+0000 7f286b59e640 1 Processor -- start 2024-03-24T10:34:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.524+0000 7f286b59e640 1 -- start start 2024-03-24T10:34:24.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286b59e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f286c15be40 0x7f286c0d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286b59e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c15c220 con 0x7f286c0d1a40 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286b59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c0b2d10 con 0x7f286c0b0fa0 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286b59e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f286c0ae860 con 0x7f286c15be40 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f286c0d1a40 0x7f286c165950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33028/0 (socket says 172.21.15.1:33028) 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286ad9d640 1 -- 172.21.15.1:0/919171385 learned_addr learned my addr 172.21.15.1:0/919171385 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286a59c640 1 --2- 172.21.15.1:0/919171385 >> v2:172.21.15.3:3300/0 conn(0x7f286c15be40 0x7f286c0d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867198439 0 0) 0x7f286c15c220 con 0x7f286c0d1a40 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2858003520 con 0x7f286c0d1a40 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455730377 0 0) 0x7f286c0b2d10 con 0x7f286c0b0fa0 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f286c15c220 con 0x7f286c0b0fa0 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286a59c640 1 -- 172.21.15.1:0/919171385 >> v1:172.21.15.3:6789/0 conn(0x7f286c0b0fa0 legacy=0x7f286c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286a59c640 1 -- 172.21.15.1:0/919171385 >> v1:172.21.15.1:6789/0 conn(0x7f286c0d1a40 legacy=0x7f286c165950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.525+0000 7f286a59c640 1 -- 172.21.15.1:0/919171385 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c0b2ff0 con 0x7f286c15be40 2024-03-24T10:34:24.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.526+0000 7f286a59c640 1 --2- 172.21.15.1:0/919171385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c15be40 0x7f286c0d05f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097a0 tx=0x7f285c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e36564d09e00224 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.526+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c031830 con 0x7f286c15be40 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.526+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c031990 con 0x7f286c15be40 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.526+0000 7f2869d9b640 1 -- 172.21.15.1:0/919171385 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c031d00 con 0x7f286c15be40 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 -- 172.21.15.1:0/919171385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c15be40 msgr2=0x7f286c0d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 --2- 172.21.15.1:0/919171385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c15be40 0x7f286c0d05f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097a0 tx=0x7f285c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 -- 172.21.15.1:0/919171385 shutdown_connections 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 --2- 172.21.15.1:0/919171385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c15be40 0x7f286c0d05f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 -- 172.21.15.1:0/919171385 >> 172.21.15.1:0/919171385 conn(0x7f286c09a9d0 msgr2=0x7f286c09ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 -- 172.21.15.1:0/919171385 shutdown_connections 2024-03-24T10:34:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.527+0000 7f286b59e640 1 -- 172.21.15.1:0/919171385 wait complete. 2024-03-24T10:34:24.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.528+0000 7f286b59e640 1 Processor -- start 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.528+0000 7f286b59e640 1 -- start start 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f286c0b0fa0 0x7f286c14f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 0x7f286c1f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0c9ef0 con 0x7f286c15be40 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0ca1d0 con 0x7f286c0d1a40 2024-03-24T10:34:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286b59e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f286c0ca4b0 con 0x7f286c0b0fa0 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34928/0 (socket says 172.21.15.1:34928) 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 -- 172.21.15.1:0/2156437126 learned_addr learned my addr 172.21.15.1:0/2156437126 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f2863fff640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 0x7f286c1f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286a59c640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f286c0b0fa0 0x7f286c14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 -- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f286c0b0fa0 msgr2=0x7f286c14f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f286c0b0fa0 0x7f286c14f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 -- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 msgr2=0x7f286c1f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 0x7f286c1f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 -- 172.21.15.1:0/2156437126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c009400 con 0x7f286c15be40 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f2863fff640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 0x7f286c1f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.529+0000 7f286ad9d640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f286400cb40 tx=0x7f286400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.530+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286400aca0 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.530+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f286400ae20 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.530+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c0ca790 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.530+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864012640 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.530+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c1f1e50 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.531+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c0cacb0 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.531+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2864007070 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.531+0000 7f2862ffd640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 0x7f284c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.531+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f286408a1a0 con 0x7f286c15be40 2024-03-24T10:34:24.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.531+0000 7f286a59c640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 0x7f284c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.532+0000 7f286a59c640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 0x7f284c06dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f285c036000 tx=0x7f285c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.536+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f286c0cacb0 con 0x7f286c15be40 2024-03-24T10:34:24.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.716+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f286c0cacb0 con 0x7f286c15be40 2024-03-24T10:34:24.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.716+0000 7f2862ffd640 1 -- 172.21.15.1:0/2156437126 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f286c0cacb0 con 0x7f286c15be40 2024-03-24T10:34:24.717 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 msgr2=0x7f284c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 0x7f284c06dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f285c036000 tx=0x7f285c008030 comp rx=0 tx=0).stop 2024-03-24T10:34:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 msgr2=0x7f286c0c6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f286400cb40 tx=0x7f286400a060 comp rx=0 tx=0).stop 2024-03-24T10:34:24.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 shutdown_connections 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f284c06b810 0x7f284c06dcd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f286c15be40 0x7f286c0c6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f286c0d1a40 0x7f286c1f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.718+0000 7f286b59e640 1 --2- 172.21.15.1:0/2156437126 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f286c0b0fa0 0x7f286c14f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.719+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 >> 172.21.15.1:0/2156437126 conn(0x7f286c09a9d0 msgr2=0x7f286c0cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.719+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 shutdown_connections 2024-03-24T10:34:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.719+0000 7f286b59e640 1 -- 172.21.15.1:0/2156437126 wait complete. 2024-03-24T10:34:24.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.834+0000 7f2751bf4640 1 Processor -- start 2024-03-24T10:34:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.834+0000 7f2751bf4640 1 -- start start 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f2751bf4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 0x7f274c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f2751bf4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f274c06f550 0x7f274c06b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f2751bf4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f274c0cf1c0 con 0x7f274c06e480 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f2751bf4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f274c06ebf0 con 0x7f274c06f550 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f2751bf4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f274c0ce8d0 con 0x7f274c15bd00 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f274c06e480 0x7f274c0cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33036/0 (socket says 172.21.15.1:33036) 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274affd640 1 -- 172.21.15.1:0/503280116 learned_addr learned my addr 172.21.15.1:0/503280116 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274bfff640 1 --2- 172.21.15.1:0/503280116 >> v2:172.21.15.3:3300/0 conn(0x7f274c06f550 0x7f274c06b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274b7fe640 1 --2- 172.21.15.1:0/503280116 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 0x7f274c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274a7fc640 1 -- 172.21.15.1:0/503280116 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756631055 0 0) 0x7f274c0cf1c0 con 0x7f274c06e480 2024-03-24T10:34:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274a7fc640 1 -- 172.21.15.1:0/503280116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2734003520 con 0x7f274c06e480 2024-03-24T10:34:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274b7fe640 1 -- 172.21.15.1:0/503280116 >> v2:172.21.15.3:3300/0 conn(0x7f274c06f550 msgr2=0x7f274c06b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.835+0000 7f274b7fe640 1 --2- 172.21.15.1:0/503280116 >> v2:172.21.15.3:3300/0 conn(0x7f274c06f550 0x7f274c06b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.836+0000 7f274b7fe640 1 -- 172.21.15.1:0/503280116 >> v1:172.21.15.1:6789/0 conn(0x7f274c06e480 legacy=0x7f274c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.836+0000 7f274b7fe640 1 -- 172.21.15.1:0/503280116 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c06e910 con 0x7f274c15bd00 2024-03-24T10:34:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.836+0000 7f274b7fe640 1 --2- 172.21.15.1:0/503280116 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 0x7f274c0cf650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f273c0097a0 tx=0x7f273c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=277aff43da2dffce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.836+0000 7f274a7fc640 1 -- 172.21.15.1:0/503280116 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c031830 con 0x7f274c15bd00 2024-03-24T10:34:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f274a7fc640 1 -- 172.21.15.1:0/503280116 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273c031990 con 0x7f274c15bd00 2024-03-24T10:34:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f274a7fc640 1 -- 172.21.15.1:0/503280116 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c031d20 con 0x7f274c15bd00 2024-03-24T10:34:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 -- 172.21.15.1:0/503280116 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 msgr2=0x7f274c0cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 --2- 172.21.15.1:0/503280116 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 0x7f274c0cf650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f273c0097a0 tx=0x7f273c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 -- 172.21.15.1:0/503280116 shutdown_connections 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 --2- 172.21.15.1:0/503280116 >> v2:172.21.15.3:3300/0 conn(0x7f274c06f550 0x7f274c06b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 --2- 172.21.15.1:0/503280116 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c15bd00 0x7f274c0cf650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 -- 172.21.15.1:0/503280116 >> 172.21.15.1:0/503280116 conn(0x7f274c0d02e0 msgr2=0x7f274c0d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 -- 172.21.15.1:0/503280116 shutdown_connections 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.837+0000 7f2751bf4640 1 -- 172.21.15.1:0/503280116 wait complete. 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.838+0000 7f2751bf4640 1 Processor -- start 2024-03-24T10:34:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.838+0000 7f2751bf4640 1 -- start start 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 0x7f274c1f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f274c153ac0 0x7f274c142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f274c14f320 con 0x7f274c15bd00 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f274c14e450 con 0x7f274c153ac0 2024-03-24T10:34:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f2751bf4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f274c1f27f0 con 0x7f274c06f550 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f274affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f274affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34942/0 (socket says 172.21.15.1:34942) 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f274affd640 1 -- 172.21.15.1:0/1344451875 learned_addr learned my addr 172.21.15.1:0/1344451875 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f274b7fe640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 0x7f274c1f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.839+0000 7f274bfff640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f274c153ac0 0x7f274c142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 -- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 msgr2=0x7f274c1f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 0x7f274c1f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 -- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f274c153ac0 msgr2=0x7f274c142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f274c153ac0 0x7f274c142820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 -- 172.21.15.1:0/1344451875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273c009400 con 0x7f274c15bd00 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274b7fe640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 0x7f274c1f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f274affd640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f274000c430 tx=0x7f274000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740005ce0 con 0x7f274c15bd00 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2740005e40 con 0x7f274c15bd00 2024-03-24T10:34:24.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274c0cfd10 con 0x7f274c15bd00 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400072f0 con 0x7f274c15bd00 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.840+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274c14e980 con 0x7f274c15bd00 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.841+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f274c04f4e0 con 0x7f274c15bd00 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.841+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2740007450 con 0x7f274c15bd00 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.842+0000 7f2733fff640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 0x7f273806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.842+0000 7f274b7fe640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 0x7f273806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:24.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.842+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2740089930 con 0x7f274c15bd00 2024-03-24T10:34:24.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.842+0000 7f274b7fe640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 0x7f273806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f273c036000 tx=0x7f273c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:24.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:24.846+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f27400542e0 con 0x7f274c15bd00 2024-03-24T10:34:25.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.025+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f274c15a020 con 0x7f274c15bd00 2024-03-24T10:34:25.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.026+0000 7f2733fff640 1 -- 172.21.15.1:0/1344451875 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f2740017020 con 0x7f274c15bd00 2024-03-24T10:34:25.026 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:25.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 msgr2=0x7f273806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 0x7f273806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f273c036000 tx=0x7f273c03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:25.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 msgr2=0x7f274c153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f274000c430 tx=0x7f274000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 shutdown_connections 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f273806b810 0x7f273806dcd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f274c153ac0 0x7f274c142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f274c15bd00 0x7f274c153580 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 --2- 172.21.15.1:0/1344451875 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f274c06f550 0x7f274c1f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 >> 172.21.15.1:0/1344451875 conn(0x7f274c0d02e0 msgr2=0x7f274c15c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 shutdown_connections 2024-03-24T10:34:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.028+0000 7f2751bf4640 1 -- 172.21.15.1:0/1344451875 wait complete. 2024-03-24T10:34:25.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-24T10:34:25.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.140+0000 7fc666215640 1 Processor -- start 2024-03-24T10:34:25.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.140+0000 7fc666215640 1 -- start start 2024-03-24T10:34:25.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc666215640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 0x7fc6600b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc666215640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc66015e770 con 0x7fc66015e390 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc666215640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6600d4e40 con 0x7fc6600d2bb0 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc666215640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc6600b3190 con 0x7fc6600d1000 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc66015e390 0x7fc6600d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33050/0 (socket says 172.21.15.1:33050) 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65f7fe640 1 -- 172.21.15.1:0/1466186482 learned_addr learned my addr 172.21.15.1:0/1466186482 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65ffff640 1 --2- 172.21.15.1:0/1466186482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 0x7fc6600b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856897984 0 0) 0x7fc650001490 con 0x7fc66015e390 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc648003520 con 0x7fc66015e390 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171488906 0 0) 0x7fc6600d4e40 con 0x7fc6600d2bb0 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.141+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc650001490 con 0x7fc6600d2bb0 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.142+0000 7fc65ffff640 1 -- 172.21.15.1:0/1466186482 >> v1:172.21.15.3:6789/0 conn(0x7fc6600d2bb0 legacy=0x7fc6600b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.142+0000 7fc65ffff640 1 -- 172.21.15.1:0/1466186482 >> v1:172.21.15.1:6789/0 conn(0x7fc66015e390 legacy=0x7fc6600d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.142+0000 7fc65ffff640 1 -- 172.21.15.1:0/1466186482 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6600d1e40 con 0x7fc6600d1000 2024-03-24T10:34:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.142+0000 7fc65ffff640 1 --2- 172.21.15.1:0/1466186482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 0x7fc6600b42a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097d0 tx=0x7fc65802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b764843ef2bd4084 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.142+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658031770 con 0x7fc6600d1000 2024-03-24T10:34:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6580318d0 con 0x7fc6600d1000 2024-03-24T10:34:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc65e7fc640 1 -- 172.21.15.1:0/1466186482 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658031c40 con 0x7fc6600d1000 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 -- 172.21.15.1:0/1466186482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 msgr2=0x7fc6600b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 --2- 172.21.15.1:0/1466186482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 0x7fc6600b42a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097d0 tx=0x7fc65802b810 comp rx=0 tx=0).stop 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 -- 172.21.15.1:0/1466186482 shutdown_connections 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 --2- 172.21.15.1:0/1466186482 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6600d1000 0x7fc6600b42a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 -- 172.21.15.1:0/1466186482 >> 172.21.15.1:0/1466186482 conn(0x7fc6600d3ec0 msgr2=0x7fc6600d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 -- 172.21.15.1:0/1466186482 shutdown_connections 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.143+0000 7fc666215640 1 -- 172.21.15.1:0/1466186482 wait complete. 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.144+0000 7fc666215640 1 Processor -- start 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.144+0000 7fc666215640 1 -- start start 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6600d1000 0x7fc6600b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc66015e390 0x7fc6601f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6600cdc30 con 0x7fc6600d2bb0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc6600b6ec0 con 0x7fc6600d1000 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc666215640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc6600b71a0 con 0x7fc66015e390 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc65effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc65effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34944/0 (socket says 172.21.15.1:34944) 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc65effd640 1 -- 172.21.15.1:0/3904025937 learned_addr learned my addr 172.21.15.1:0/3904025937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc65f7fe640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6600d1000 0x7fc6600b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.145+0000 7fc65ffff640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc66015e390 0x7fc6601f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 -- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc66015e390 msgr2=0x7fc6601f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc66015e390 0x7fc6601f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 -- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6600d1000 msgr2=0x7fc6600b8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6600d1000 0x7fc6600b8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 -- 172.21.15.1:0/3904025937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc658009400 con 0x7fc6600d2bb0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc65effd640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c500 tx=0x7fc65400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540052d0 con 0x7fc6600d2bb0 2024-03-24T10:34:25.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc654005450 con 0x7fc6600d2bb0 2024-03-24T10:34:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6600b7480 con 0x7fc6600d2bb0 2024-03-24T10:34:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654012870 con 0x7fc6600d2bb0 2024-03-24T10:34:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.146+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6600b7a40 con 0x7fc6600d2bb0 2024-03-24T10:34:25.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.147+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66004f4e0 con 0x7fc6600d2bb0 2024-03-24T10:34:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.147+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc6540129d0 con 0x7fc6600d2bb0 2024-03-24T10:34:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.148+0000 7fc647fff640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 0x7fc64c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.148+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc654089cc0 con 0x7fc6600d2bb0 2024-03-24T10:34:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.148+0000 7fc65f7fe640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 0x7fc64c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.148+0000 7fc65f7fe640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 0x7fc64c06dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc6500016f0 tx=0x7fc650001b80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:25.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.152+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6540547a0 con 0x7fc6600d2bb0 2024-03-24T10:34:25.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:25.402+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7fc6600d2280 con 0x7fc6600d2bb0 2024-03-24T10:34:25.900 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:34:25.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:34:26.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.227+0000 7fc647fff640 1 -- 172.21.15.1:0/3904025937 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v55) ==== 150+0+0 (secure 0 0 0) 0x7fc654057e60 con 0x7fc6600d2bb0 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 msgr2=0x7fc64c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 0x7fc64c06dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc6500016f0 tx=0x7fc650001b80 comp rx=0 tx=0).stop 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 msgr2=0x7fc6600b9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c500 tx=0x7fc65400c9c0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 shutdown_connections 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc64c06b810 0x7fc64c06dcd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc66015e390 0x7fc6601f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6600d2bb0 0x7fc6600b9190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 --2- 172.21.15.1:0/3904025937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6600d1000 0x7fc6600b8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.230+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 >> 172.21.15.1:0/3904025937 conn(0x7fc6600d3ec0 msgr2=0x7fc6600cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.231+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 shutdown_connections 2024-03-24T10:34:26.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.231+0000 7fc666215640 1 -- 172.21.15.1:0/3904025937 wait complete. 2024-03-24T10:34:26.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.355+0000 7fd40b2d4640 1 Processor -- start 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.355+0000 7fd40b2d4640 1 -- start start 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd40b2d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd404144550 0x7fd4040b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd40b2d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd40406c930 con 0x7fd404071cc0 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd40b2d4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd40406cb80 con 0x7fd4040b3a60 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd40b2d4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd4040b4310 con 0x7fd404144550 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd409070640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd404144550 0x7fd4040b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd409070640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd404144550 0x7fd4040b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34960/0 (socket says 172.21.15.1:34960) 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd409070640 1 -- 172.21.15.1:0/98379312 learned_addr learned my addr 172.21.15.1:0/98379312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.356+0000 7fd409871640 1 --1- 172.21.15.1:0/98379312 >> v1:172.21.15.1:6789/0 conn(0x7fd404071cc0 0x7fd40406dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33066/0 (socket says 172.21.15.1:33066) 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.357+0000 7fd409070640 1 -- 172.21.15.1:0/98379312 >> v1:172.21.15.3:6789/0 conn(0x7fd4040b3a60 legacy=0x7fd4040ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.357+0000 7fd409070640 1 -- 172.21.15.1:0/98379312 >> v1:172.21.15.1:6789/0 conn(0x7fd404071cc0 legacy=0x7fd40406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.357+0000 7fd3fbfff640 1 -- 172.21.15.1:0/98379312 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971104714 0 0) 0x7fd40406cb80 con 0x7fd4040b3a60 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.357+0000 7fd409070640 1 -- 172.21.15.1:0/98379312 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40406ce60 con 0x7fd404144550 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.357+0000 7fd409070640 1 --2- 172.21.15.1:0/98379312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4040b3520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00bb10 tx=0x7fd3fc02dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641f8d4f9670366d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd3fbfff640 1 -- 172.21.15.1:0/98379312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc02ecd0 con 0x7fd404144550 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd3fbfff640 1 -- 172.21.15.1:0/98379312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3fc02ee30 con 0x7fd404144550 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd3fbfff640 1 -- 172.21.15.1:0/98379312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc036660 con 0x7fd404144550 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 -- 172.21.15.1:0/98379312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 msgr2=0x7fd4040b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/98379312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4040b3520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00bb10 tx=0x7fd3fc02dfa0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 -- 172.21.15.1:0/98379312 shutdown_connections 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/98379312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4040b3520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 -- 172.21.15.1:0/98379312 >> 172.21.15.1:0/98379312 conn(0x7fd40406b4f0 msgr2=0x7fd40406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 -- 172.21.15.1:0/98379312 shutdown_connections 2024-03-24T10:34:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.358+0000 7fd40b2d4640 1 -- 172.21.15.1:0/98379312 wait complete. 2024-03-24T10:34:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.359+0000 7fd40b2d4640 1 Processor -- start 2024-03-24T10:34:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.359+0000 7fd40b2d4640 1 -- start start 2024-03-24T10:34:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd404071cc0 0x7fd40415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4040b3a60 0x7fd4041f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd40415aa40 con 0x7fd404144550 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041ef860 con 0x7fd4040b3a60 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40b2d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd4041ef560 con 0x7fd404071cc0 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34974/0 (socket says 172.21.15.1:34974) 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 -- 172.21.15.1:0/3455115828 learned_addr learned my addr 172.21.15.1:0/3455115828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd40886f640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4040b3a60 0x7fd4041f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409070640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd404071cc0 0x7fd40415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 -- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd404071cc0 msgr2=0x7fd40415a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd404071cc0 0x7fd40415a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 -- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4040b3a60 msgr2=0x7fd4041f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.360+0000 7fd409871640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4040b3a60 0x7fd4041f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd409871640 1 -- 172.21.15.1:0/3455115828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc00b770 con 0x7fd404144550 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd409871640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd40000cb20 tx=0x7fd4000020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40000a760 con 0x7fd404144550 2024-03-24T10:34:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd40000a8c0 con 0x7fd404144550 2024-03-24T10:34:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40000ac50 con 0x7fd404144550 2024-03-24T10:34:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4041f3270 con 0x7fd404144550 2024-03-24T10:34:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.361+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4041f2f80 con 0x7fd404144550 2024-03-24T10:34:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.362+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4041efa10 con 0x7fd404144550 2024-03-24T10:34:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.362+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4000070d0 con 0x7fd404144550 2024-03-24T10:34:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.363+0000 7fd3f9ffb640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 0x7fd3e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.363+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd400089ad0 con 0x7fd404144550 2024-03-24T10:34:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.363+0000 7fd409070640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 0x7fd3e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.363+0000 7fd409070640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 0x7fd3e806dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc038000 tx=0x7fd3fc032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.367+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4041efa10 con 0x7fd404144550 2024-03-24T10:34:26.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.550+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd4041efa10 con 0x7fd404144550 2024-03-24T10:34:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.552+0000 7fd3f9ffb640 1 -- 172.21.15.1:0/3455115828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fd4041efa10 con 0x7fd404144550 2024-03-24T10:34:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 msgr2=0x7fd3e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 0x7fd3e806dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc038000 tx=0x7fd3fc032000 comp rx=0 tx=0).stop 2024-03-24T10:34:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 msgr2=0x7fd4041efcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd40000cb20 tx=0x7fd4000020f0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 shutdown_connections 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3e806b810 0x7fd3e806dcd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd404144550 0x7fd4041efcf0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4040b3a60 0x7fd4041f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 --2- 172.21.15.1:0/3455115828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd404071cc0 0x7fd40415a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 >> 172.21.15.1:0/3455115828 conn(0x7fd40406b4f0 msgr2=0x7fd4040cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 shutdown_connections 2024-03-24T10:34:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.555+0000 7fd40b2d4640 1 -- 172.21.15.1:0/3455115828 wait complete. 2024-03-24T10:34:26.566 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T10:34:26.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:34:26.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:34:26.656 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T10:34:26.657 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:26.657 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:26.672 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:26.672 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:26.728 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.870+0000 7fbe5c1b5640 1 Processor -- start 2024-03-24T10:34:26.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.870+0000 7fbe5c1b5640 1 -- start start 2024-03-24T10:34:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5c1b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5c1b5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 0x7fbe540adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5c1b5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe541553c0 con 0x7fbe540ae1b0 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5c1b5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5406d250 con 0x7fbe5406b5e0 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5c1b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbe540ae640 con 0x7fbe54154fe0 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59750640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 0x7fbe540adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5a752640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbe540ae1b0 0x7fbe54161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49324/0 (socket says 172.21.15.1:49324) 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:39170/0 (socket says 172.21.15.1:39170) 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe5a752640 1 -- 172.21.15.1:0/1247179119 learned_addr learned my addr 172.21.15.1:0/1247179119 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 -- 172.21.15.1:0/1247179119 >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 msgr2=0x7fbe540adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 --2- 172.21.15.1:0/1247179119 >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 0x7fbe540adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 -- 172.21.15.1:0/1247179119 >> v1:172.21.15.3:6789/0 conn(0x7fbe540ae1b0 legacy=0x7fbe54161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59f51640 1 -- 172.21.15.1:0/1247179119 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5406c7e0 con 0x7fbe54154fe0 2024-03-24T10:34:26.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.871+0000 7fbe59750640 1 --2- 172.21.15.1:0/1247179119 >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 0x7fbe540adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.872+0000 7fbe59f51640 1 --2- 172.21.15.1:0/1247179119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe500097a0 tx=0x7fbe5002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0de21d82b8c0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.872+0000 7fbe58f4f640 1 -- 172.21.15.1:0/1247179119 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50031830 con 0x7fbe54154fe0 2024-03-24T10:34:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.872+0000 7fbe58f4f640 1 -- 172.21.15.1:0/1247179119 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe50031990 con 0x7fbe54154fe0 2024-03-24T10:34:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.872+0000 7fbe58f4f640 1 -- 172.21.15.1:0/1247179119 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe50031d20 con 0x7fbe54154fe0 2024-03-24T10:34:26.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1247179119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 msgr2=0x7fbe5406b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1247179119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe500097a0 tx=0x7fbe5002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1247179119 shutdown_connections 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1247179119 >> v2:172.21.15.3:3300/0 conn(0x7fbe5406b5e0 0x7fbe540adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1247179119 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe5406b0a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1247179119 >> 172.21.15.1:0/1247179119 conn(0x7fbe540d0570 msgr2=0x7fbe540d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1247179119 shutdown_connections 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.873+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1247179119 wait complete. 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.874+0000 7fbe5c1b5640 1 Processor -- start 2024-03-24T10:34:26.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.874+0000 7fbe5c1b5640 1 -- start start 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe5406b5e0 0x7fbe541e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe541539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbe54158c20 con 0x7fbe540ae1b0 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5415de10 con 0x7fbe5406b5e0 2024-03-24T10:34:26.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5c1b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbe541e7eb0 con 0x7fbe54154fe0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34984/0 (socket says 172.21.15.1:34984) 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 -- 172.21.15.1:0/1145961183 learned_addr learned my addr 172.21.15.1:0/1145961183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59f51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe5406b5e0 0x7fbe541e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe5a752640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe541539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 -- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 msgr2=0x7fbe541539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe541539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 -- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe5406b5e0 msgr2=0x7fbe541e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe5406b5e0 0x7fbe541e78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 -- 172.21.15.1:0/1145961183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe44009400 con 0x7fbe540ae1b0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.875+0000 7fbe59750640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400ebe0 tx=0x7fbe4400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.876+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44009760 con 0x7fbe540ae1b0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.876+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4400cc30 con 0x7fbe540ae1b0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.876+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe50009400 con 0x7fbe540ae1b0 2024-03-24T10:34:26.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.876+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4400eee0 con 0x7fbe540ae1b0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.876+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5415eb20 con 0x7fbe540ae1b0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.877+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe5406c430 con 0x7fbe540ae1b0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.877+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbe44013470 con 0x7fbe540ae1b0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.878+0000 7fbe4a7fc640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 0x7fbe3806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.878+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe4408a6b0 con 0x7fbe540ae1b0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.878+0000 7fbe59f51640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 0x7fbe3806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:26.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.878+0000 7fbe59f51640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 0x7fbe3806dd00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbe50036000 tx=0x7fbe5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:26.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:26.882+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbe44055210 con 0x7fbe540ae1b0 2024-03-24T10:34:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.062+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe5404f4e0 con 0x7fbe540ae1b0 2024-03-24T10:34:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.063+0000 7fbe4a7fc640 1 -- 172.21.15.1:0/1145961183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fbe440588d0 con 0x7fbe540ae1b0 2024-03-24T10:34:27.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:27.064 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 msgr2=0x7fbe3806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 0x7fbe3806dd00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbe50036000 tx=0x7fbe5003c040 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 msgr2=0x7fbe5415d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400ebe0 tx=0x7fbe4400c520 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 shutdown_connections 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe3806b840 0x7fbe3806dd00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe54154fe0 0x7fbe541539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe540ae1b0 0x7fbe5415d8d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 --2- 172.21.15.1:0/1145961183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe5406b5e0 0x7fbe541e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 >> 172.21.15.1:0/1145961183 conn(0x7fbe540d0570 msgr2=0x7fbe540d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 shutdown_connections 2024-03-24T10:34:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.065+0000 7fbe5c1b5640 1 -- 172.21.15.1:0/1145961183 wait complete. 2024-03-24T10:34:27.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:27.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:27.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.183+0000 7f5e6b2b5640 1 Processor -- start 2024-03-24T10:34:27.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.183+0000 7f5e6b2b5640 1 -- start start 2024-03-24T10:34:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e6b2b5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5e64158540 0x7f5e6406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e6b2b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e640ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e6b2b5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e64072760 con 0x7f5e6406b5e0 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e6b2b5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6406ba60 con 0x7f5e64158540 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e6b2b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5e640b05d0 con 0x7f5e640b0100 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69852640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e640ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69051640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5e64158540 0x7f5e6406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e68850640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5e6406b5e0 0x7f5e640af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49326/0 (socket says 172.21.15.1:49326) 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e68850640 1 -- 172.21.15.1:0/987917099 learned_addr learned my addr 172.21.15.1:0/987917099 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e5bfff640 1 -- 172.21.15.1:0/987917099 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 427643574 0 0) 0x7f5e64072760 con 0x7f5e6406b5e0 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e5bfff640 1 -- 172.21.15.1:0/987917099 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e50003520 con 0x7f5e6406b5e0 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69051640 1 -- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 msgr2=0x7f5e640ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69051640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e640ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69051640 1 -- 172.21.15.1:0/987917099 >> v1:172.21.15.3:6789/0 conn(0x7f5e6406b5e0 legacy=0x7f5e640af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69051640 1 -- 172.21.15.1:0/987917099 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6415f510 con 0x7f5e64158540 2024-03-24T10:34:27.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.184+0000 7f5e69852640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e640ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.185+0000 7f5e69051640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e64158540 0x7f5e6406b0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e600097a0 tx=0x7f5e6002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f63f00636cca1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.185+0000 7f5e5bfff640 1 -- 172.21.15.1:0/987917099 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002e070 con 0x7f5e64158540 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.185+0000 7f5e5bfff640 1 -- 172.21.15.1:0/987917099 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e600315a0 con 0x7f5e64158540 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.185+0000 7f5e5bfff640 1 -- 172.21.15.1:0/987917099 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60031910 con 0x7f5e64158540 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e64158540 msgr2=0x7f5e6406b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e64158540 0x7f5e6406b0a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e600097a0 tx=0x7f5e6002b840 comp rx=0 tx=0).stop 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/987917099 shutdown_connections 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e640ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/987917099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e64158540 0x7f5e6406b0a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/987917099 >> 172.21.15.1:0/987917099 conn(0x7f5e640b2820 msgr2=0x7f5e640b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/987917099 shutdown_connections 2024-03-24T10:34:27.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.186+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/987917099 wait complete. 2024-03-24T10:34:27.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.187+0000 7f5e6b2b5640 1 Processor -- start 2024-03-24T10:34:27.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.187+0000 7f5e6b2b5640 1 -- start start 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 0x7f5e641f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e64154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6414d640 con 0x7f5e64158540 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5e64155130 con 0x7f5e6406b5e0 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e6b2b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6414d890 con 0x7f5e640b0100 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34988/0 (socket says 172.21.15.1:34988) 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69051640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 0x7f5e641f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 -- 172.21.15.1:0/721177240 learned_addr learned my addr 172.21.15.1:0/721177240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e68850640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e64154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 -- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 msgr2=0x7f5e64154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e64154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 -- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 msgr2=0x7f5e641f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 0x7f5e641f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 -- 172.21.15.1:0/721177240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5c009a70 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69051640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 0x7f5e641f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.188+0000 7f5e69852640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00efb0 tx=0x7f5e5c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.189+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c009d50 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.189+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e5c00f040 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.189+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e60009400 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.189+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c0146a0 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.189+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6414de50 con 0x7f5e64158540 2024-03-24T10:34:27.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.190+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e64155390 con 0x7f5e64158540 2024-03-24T10:34:27.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.190+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e5c014800 con 0x7f5e64158540 2024-03-24T10:34:27.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.190+0000 7f5e59ffb640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 0x7f5e4806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.191+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e5c08b050 con 0x7f5e64158540 2024-03-24T10:34:27.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.191+0000 7f5e69051640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 0x7f5e4806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.191+0000 7f5e69051640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 0x7f5e4806dd00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e60036000 tx=0x7f5e6003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.195+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e5c055b20 con 0x7f5e64158540 2024-03-24T10:34:27.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.374+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e6404f4e0 con 0x7f5e64158540 2024-03-24T10:34:27.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.375+0000 7f5e59ffb640 1 -- 172.21.15.1:0/721177240 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f5e5c0180a0 con 0x7f5e64158540 2024-03-24T10:34:27.375 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 msgr2=0x7f5e4806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 0x7f5e4806dd00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e60036000 tx=0x7f5e6003d040 comp rx=0 tx=0).stop 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 msgr2=0x7f5e6414e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00efb0 tx=0x7f5e5c00c490 comp rx=0 tx=0).stop 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 shutdown_connections 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5e4806b840 0x7f5e4806dd00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e64158540 0x7f5e6414e8f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5e640b0100 0x7f5e64154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 --2- 172.21.15.1:0/721177240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e6406b5e0 0x7f5e641f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.376+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 >> 172.21.15.1:0/721177240 conn(0x7f5e640b2820 msgr2=0x7f5e640af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.377+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 shutdown_connections 2024-03-24T10:34:27.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.377+0000 7f5e6b2b5640 1 -- 172.21.15.1:0/721177240 wait complete. 2024-03-24T10:34:27.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 Processor -- start 2024-03-24T10:34:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 -- start start 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8d080b31d0 0x7f8d080d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 0x7f8d080ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d080cc250 con 0x7f8d080d0e90 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f8d080b35b0 con 0x7f8d080b31d0 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.492+0000 7f8d0f0c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8d080cc4a0 con 0x7f8d0806d240 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d07fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8d080d0e90 0x7f8d0806cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33078/0 (socket says 172.21.15.1:33078) 2024-03-24T10:34:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d07fff640 1 -- 172.21.15.1:0/3519928108 learned_addr learned my addr 172.21.15.1:0/3519928108 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/3519928108 >> v2:172.21.15.3:3300/0 conn(0x7f8d080b31d0 0x7f8d080d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0d65d640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 0x7f8d080ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d077fe640 1 -- 172.21.15.1:0/3519928108 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688064905 0 0) 0x7f8d080cc250 con 0x7f8d080d0e90 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d077fe640 1 -- 172.21.15.1:0/3519928108 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf8003520 con 0x7f8d080d0e90 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0ce5c640 1 -- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 msgr2=0x7f8d080ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 0x7f8d080ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0ce5c640 1 -- 172.21.15.1:0/3519928108 >> v1:172.21.15.1:6789/0 conn(0x7f8d080d0e90 legacy=0x7f8d0806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0ce5c640 1 -- 172.21.15.1:0/3519928108 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d080cc780 con 0x7f8d080b31d0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.493+0000 7f8d0d65d640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 0x7f8d080ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d080b31d0 0x7f8d080d0950 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0097a0 tx=0x7f8cfc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eb70cf6cc8dee9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d077fe640 1 -- 172.21.15.1:0/3519928108 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc031830 con 0x7f8d080b31d0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d077fe640 1 -- 172.21.15.1:0/3519928108 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cfc031990 con 0x7f8d080b31d0 2024-03-24T10:34:27.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d077fe640 1 -- 172.21.15.1:0/3519928108 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc031d00 con 0x7f8d080b31d0 2024-03-24T10:34:27.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d080b31d0 msgr2=0x7f8d080d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d080b31d0 0x7f8d080d0950 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0097a0 tx=0x7f8cfc02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/3519928108 shutdown_connections 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d0806d240 0x7f8d080ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/3519928108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d080b31d0 0x7f8d080d0950 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/3519928108 >> 172.21.15.1:0/3519928108 conn(0x7f8d080d1c00 msgr2=0x7f8d080d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/3519928108 shutdown_connections 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.494+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/3519928108 wait complete. 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 Processor -- start 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 -- start start 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d0806d240 0x7f8d0815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d080d0e90 0x7f8d0815cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8d080c58d0 con 0x7f8d080b31d0 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8d080c5bb0 con 0x7f8d0806d240 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.496+0000 7f8d0f0c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8d080c5e90 con 0x7f8d080d0e90 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34998/0 (socket says 172.21.15.1:34998) 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 -- 172.21.15.1:0/1893818057 learned_addr learned my addr 172.21.15.1:0/1893818057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d0806d240 0x7f8d0815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d0d65d640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d080d0e90 0x7f8d0815cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 -- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d080d0e90 msgr2=0x7f8d0815cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d080d0e90 0x7f8d0815cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 -- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d0806d240 msgr2=0x7f8d0815ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d0806d240 0x7f8d0815ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 -- 172.21.15.1:0/1893818057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d00009a70 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d07fff640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4001eb0 tx=0x7f8cf400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4005ce0 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf4005e60 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cfc009400 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40072f0 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.497+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d080c6450 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.498+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0804f470 con 0x7f8d080b31d0 2024-03-24T10:34:27.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.499+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8cf4007450 con 0x7f8d080b31d0 2024-03-24T10:34:27.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.499+0000 7f8d057fa640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.499+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8cf4089650 con 0x7f8d080b31d0 2024-03-24T10:34:27.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.499+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.499+0000 7f8d0ce5c640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc036000 tx=0x7f8cfc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.503+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8cf4054120 con 0x7f8d080b31d0 2024-03-24T10:34:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.682+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8d080b1fd0 con 0x7f8d080b31d0 2024-03-24T10:34:27.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.683+0000 7f8d057fa640 1 -- 172.21.15.1:0/1893818057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f8cf40577e0 con 0x7f8d080b31d0 2024-03-24T10:34:27.683 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 msgr2=0x7f8cec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc036000 tx=0x7f8cfc03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 msgr2=0x7f8d080b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4001eb0 tx=0x7f8cf400c760 comp rx=0 tx=0).stop 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 shutdown_connections 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8d080d0e90 0x7f8d0815cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d080b31d0 0x7f8d080b66e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 --2- 172.21.15.1:0/1893818057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d0806d240 0x7f8d0815ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 >> 172.21.15.1:0/1893818057 conn(0x7f8d080d1c00 msgr2=0x7f8d0806ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 shutdown_connections 2024-03-24T10:34:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.685+0000 7f8d0f0c0640 1 -- 172.21.15.1:0/1893818057 wait complete. 2024-03-24T10:34:27.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:27.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.802+0000 7fa19f9c9640 1 Processor -- start 2024-03-24T10:34:27.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.802+0000 7fa19f9c9640 1 -- start start 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19f9c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa198072910 0x7fa19806b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19f9c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1980cfb40 con 0x7fa198145720 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19f9c9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1980cfe20 con 0x7fa19806b5e0 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19f9c9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa1980cf940 con 0x7fa198072910 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19cf64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa198072910 0x7fa19806b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19d765640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa198145720 0x7fa1980cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33084/0 (socket says 172.21.15.1:33084) 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19d765640 1 -- 172.21.15.1:0/1579936224 learned_addr learned my addr 172.21.15.1:0/1579936224 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19cf64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa198072910 0x7fa19806b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35000/0 (socket says 172.21.15.1:35000) 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19cf64640 1 -- 172.21.15.1:0/1579936224 >> v1:172.21.15.3:6789/0 conn(0x7fa19806b5e0 legacy=0x7fa19806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19cf64640 1 -- 172.21.15.1:0/1579936224 >> v1:172.21.15.1:6789/0 conn(0x7fa198145720 legacy=0x7fa1980cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa18ffff640 1 -- 172.21.15.1:0/1579936224 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839558589 0 0) 0x7fa1980cfb40 con 0x7fa198145720 2024-03-24T10:34:27.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.803+0000 7fa19cf64640 1 -- 172.21.15.1:0/1579936224 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1980d0100 con 0x7fa198072910 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.804+0000 7fa19cf64640 1 --2- 172.21.15.1:0/1579936224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa19806b0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa1880097a0 tx=0x7fa18802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511124f834e0f170 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.804+0000 7fa18ffff640 1 -- 172.21.15.1:0/1579936224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188030610 con 0x7fa198072910 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.804+0000 7fa18ffff640 1 -- 172.21.15.1:0/1579936224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa188030770 con 0x7fa198072910 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.804+0000 7fa18ffff640 1 -- 172.21.15.1:0/1579936224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188030b00 con 0x7fa198072910 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 -- 172.21.15.1:0/1579936224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 msgr2=0x7fa19806b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/1579936224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa19806b0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa1880097a0 tx=0x7fa18802b960 comp rx=0 tx=0).stop 2024-03-24T10:34:27.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 -- 172.21.15.1:0/1579936224 shutdown_connections 2024-03-24T10:34:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/1579936224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa19806b0a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 -- 172.21.15.1:0/1579936224 >> 172.21.15.1:0/1579936224 conn(0x7fa1980b2820 msgr2=0x7fa1980b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 -- 172.21.15.1:0/1579936224 shutdown_connections 2024-03-24T10:34:27.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.805+0000 7fa19f9c9640 1 -- 172.21.15.1:0/1579936224 wait complete. 2024-03-24T10:34:27.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.806+0000 7fa19f9c9640 1 Processor -- start 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.806+0000 7fa19f9c9640 1 -- start start 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa19806b5e0 0x7fa1980b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa198145720 0x7fa19813f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa1980ae950 con 0x7fa198072910 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa1981f6de0 con 0x7fa19806b5e0 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa1980b5080 con 0x7fa198145720 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35014/0 (socket says 172.21.15.1:35014) 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 -- 172.21.15.1:0/2205618115 learned_addr learned my addr 172.21.15.1:0/2205618115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19df66640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa198145720 0x7fa19813f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19d765640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa19806b5e0 0x7fa1980b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 -- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa198145720 msgr2=0x7fa19813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa198145720 0x7fa19813f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 -- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa19806b5e0 msgr2=0x7fa1980b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa19806b5e0 0x7fa1980b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 -- 172.21.15.1:0/2205618115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa188009400 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19cf64640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa188009890 tx=0x7fa188004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18803e040 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa188039ce0 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19813ba10 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.807+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188043680 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.808+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1980ae020 con 0x7fa198072910 2024-03-24T10:34:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.809+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198068680 con 0x7fa198072910 2024-03-24T10:34:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.809+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa1880423e0 con 0x7fa198072910 2024-03-24T10:34:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.809+0000 7fa18dffb640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 0x7fa17c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.809+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1880ac670 con 0x7fa198072910 2024-03-24T10:34:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.809+0000 7fa19d765640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 0x7fa17c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.810+0000 7fa19d765640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 0x7fa17c06dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa194000f60 tx=0x7fa194007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:27.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.814+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa188077010 con 0x7fa198072910 2024-03-24T10:34:27.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.993+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa19804f4e0 con 0x7fa198072910 2024-03-24T10:34:27.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.994+0000 7fa18dffb640 1 -- 172.21.15.1:0/2205618115 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fa18807a6d0 con 0x7fa198072910 2024-03-24T10:34:27.994 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.995+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 msgr2=0x7fa17c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.995+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 0x7fa17c06dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa194000f60 tx=0x7fa194007e40 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.995+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 msgr2=0x7fa1981f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa188009890 tx=0x7fa188004730 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 shutdown_connections 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa17c06b810 0x7fa17c06dcd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa198145720 0x7fa19813f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa198072910 0x7fa1981f68a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 --2- 172.21.15.1:0/2205618115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa19806b5e0 0x7fa1980b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 >> 172.21.15.1:0/2205618115 conn(0x7fa1980b2820 msgr2=0x7fa1980cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 shutdown_connections 2024-03-24T10:34:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:27.996+0000 7fa19f9c9640 1 -- 172.21.15.1:0/2205618115 wait complete. 2024-03-24T10:34:28.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.109+0000 7ff5e948f640 1 Processor -- start 2024-03-24T10:34:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.109+0000 7ff5e948f640 1 -- start start 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e948f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5e415be90 0x7ff5e40d2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e948f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e948f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e40cda80 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e948f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e4160430 con 0x7ff5e415be90 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e948f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e40cf850 con 0x7ff5e415fa60 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5e415be90 0x7ff5e40d2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e27fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5e40cd2a0 0x7ff5e40d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33092/0 (socket says 172.21.15.1:33092) 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e27fc640 1 -- 172.21.15.1:0/3506465352 learned_addr learned my addr 172.21.15.1:0/3506465352 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5e415be90 0x7ff5e40d2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35016/0 (socket says 172.21.15.1:35016) 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e37fe640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3506465352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 msgr2=0x7ff5e415fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3506465352 >> v1:172.21.15.1:6789/0 conn(0x7ff5e40cd2a0 legacy=0x7ff5e40d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e1ffb640 1 -- 172.21.15.1:0/3506465352 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122687205 0 0) 0x7ff5e40cda80 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.110+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3506465352 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e40cfb30 con 0x7ff5e415be90 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.111+0000 7ff5e2ffd640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e415be90 0x7ff5e40d2810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009d90 tx=0x7ff5d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3e5d50fdc924b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.111+0000 7ff5e1ffb640 1 -- 172.21.15.1:0/3506465352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d402cc50 con 0x7ff5e415be90 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.111+0000 7ff5e1ffb640 1 -- 172.21.15.1:0/3506465352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d402cdb0 con 0x7ff5e415be90 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.111+0000 7ff5e1ffb640 1 -- 172.21.15.1:0/3506465352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4034660 con 0x7ff5e415be90 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 -- 172.21.15.1:0/3506465352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e415be90 msgr2=0x7ff5e40d2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e415be90 0x7ff5e40d2810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009d90 tx=0x7ff5d402bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 -- 172.21.15.1:0/3506465352 shutdown_connections 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3506465352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e415be90 0x7ff5e40d2810 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 -- 172.21.15.1:0/3506465352 >> 172.21.15.1:0/3506465352 conn(0x7ff5e40d1c80 msgr2=0x7ff5e40d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 -- 172.21.15.1:0/3506465352 shutdown_connections 2024-03-24T10:34:28.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.112+0000 7ff5e948f640 1 -- 172.21.15.1:0/3506465352 wait complete. 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.113+0000 7ff5e948f640 1 Processor -- start 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.113+0000 7ff5e948f640 1 -- start start 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e415be90 0x7ff5e415c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e40c9440 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e40c9ca0 con 0x7ff5e415be90 2024-03-24T10:34:28.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e948f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e415f240 con 0x7ff5e415fa60 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35024/0 (socket says 172.21.15.1:35024) 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3156642258 learned_addr learned my addr 172.21.15.1:0/3156642258 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e27fc640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e415be90 0x7ff5e415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e37fe640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 msgr2=0x7ff5e415ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e415be90 msgr2=0x7ff5e415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e415be90 0x7ff5e415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.114+0000 7ff5e2ffd640 1 -- 172.21.15.1:0/3156642258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d4009a70 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5e2ffd640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009d90 tx=0x7ff5d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4040650 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d40407b0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4040b60 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e40c9650 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.115+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e41486c0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.116+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e40cc140 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.116+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5d4040cc0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.117+0000 7ff5db7fe640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 0x7ff5c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.117+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5d40acef0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.117+0000 7ff5e27fc640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 0x7ff5c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.117+0000 7ff5e27fc640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 0x7ff5c406dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4068490 tx=0x7ff5cc007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.121+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5d4077ad0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.315+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff5e40cefe0 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.315+0000 7ff5db7fe640 1 -- 172.21.15.1:0/3156642258 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7ff5d407b190 con 0x7ff5e40cd2a0 2024-03-24T10:34:28.316 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 msgr2=0x7ff5c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 0x7ff5c406dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4068490 tx=0x7ff5cc007d50 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 msgr2=0x7ff5e41f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009d90 tx=0x7ff5d4004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 shutdown_connections 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff5c406b810 0x7ff5c406dcd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5e415fa60 0x7ff5e415ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e415be90 0x7ff5e415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 --2- 172.21.15.1:0/3156642258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e40cd2a0 0x7ff5e41f3b70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 >> 172.21.15.1:0/3156642258 conn(0x7ff5e40d1c80 msgr2=0x7ff5e40cdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 shutdown_connections 2024-03-24T10:34:28.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.317+0000 7ff5e948f640 1 -- 172.21.15.1:0/3156642258 wait complete. 2024-03-24T10:34:28.329 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:28.329 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:28.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.485+0000 7fc6f2d2c640 1 Processor -- start 2024-03-24T10:34:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.485+0000 7fc6f2d2c640 1 -- start start 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec15bda0 0x7fc6ec15ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 0x7fc6ec0cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6ec0cff00 0x7fc6ec071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ec0b2db0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ec06e140 con 0x7fc6ec06c910 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f2d2c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec0d0390 con 0x7fc6ec15bda0 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6ec0cff00 0x7fc6ec071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6ec0cff00 0x7fc6ec071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35028/0 (socket says 172.21.15.1:35028) 2024-03-24T10:34:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 -- 172.21.15.1:0/3908245994 learned_addr learned my addr 172.21.15.1:0/3908245994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f0ac8640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec15bda0 0x7fc6ec15ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/3908245994 >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 0x7fc6ec0cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 -- 172.21.15.1:0/3908245994 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec15bda0 msgr2=0x7fc6ec15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec15bda0 0x7fc6ec15ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 -- 172.21.15.1:0/3908245994 >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 msgr2=0x7fc6ec0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 --2- 172.21.15.1:0/3908245994 >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 0x7fc6ec0cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6f12c9640 1 -- 172.21.15.1:0/3908245994 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec15d3c0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.486+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/3908245994 >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 0x7fc6ec0cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.487+0000 7fc6f12c9640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec0cff00 0x7fc6ec071980 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec073500 tx=0x7fc6d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5443be8d076889 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.487+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/3908245994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40317c0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.487+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/3908245994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d4031920 con 0x7fc6ec0cff00 2024-03-24T10:34:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.487+0000 7fc6eb7fe640 1 -- 172.21.15.1:0/3908245994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4031cb0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/3908245994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec0cff00 msgr2=0x7fc6ec071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec0cff00 0x7fc6ec071980 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec073500 tx=0x7fc6d402b860 comp rx=0 tx=0).stop 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/3908245994 shutdown_connections 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec0cff00 0x7fc6ec071980 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/3908245994 >> v2:172.21.15.3:3300/0 conn(0x7fc6ec06c910 0x7fc6ec0cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/3908245994 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec15bda0 0x7fc6ec15ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/3908245994 >> 172.21.15.1:0/3908245994 conn(0x7fc6ec0b17a0 msgr2=0x7fc6ec0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/3908245994 shutdown_connections 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.488+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/3908245994 wait complete. 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.489+0000 7fc6f2d2c640 1 Processor -- start 2024-03-24T10:34:28.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.489+0000 7fc6f2d2c640 1 -- start start 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 0x7fc6ec154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1f53e0 con 0x7fc6ec15bda0 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1f56c0 con 0x7fc6ec06c910 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f2d2c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec1f59a0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f12c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f0ac8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 0x7fc6ec154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:39240/0 (socket says 172.21.15.1:39240) 2024-03-24T10:34:28.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f12c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35042/0 (socket says 172.21.15.1:35042) 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6ebfff640 1 -- 172.21.15.1:0/1333988124 learned_addr learned my addr 172.21.15.1:0/1333988124 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.490+0000 7fc6f0ac8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 0x7fc6ec154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 -- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 msgr2=0x7fc6ec154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 0x7fc6ec154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 -- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 msgr2=0x7fc6ec1f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 -- 172.21.15.1:0/1333988124 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6dc009a70 con 0x7fc6ec0cff00 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6f12c9640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6ebfff640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00efc0 tx=0x7fc6dc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc009d70 con 0x7fc6ec0cff00 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00f040 con 0x7fc6ec0cff00 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0148c0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d40094d0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.491+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1f50f0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.492+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec04f4e0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.492+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc6dc014a20 con 0x7fc6ec0cff00 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.493+0000 7fc6e97fa640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 0x7fc6cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.493+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc6dc08ae60 con 0x7fc6ec0cff00 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.493+0000 7fc6f0ac8640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 0x7fc6cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.493+0000 7fc6f0ac8640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 0x7fc6cc06dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009a20 tx=0x7fc6e4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.497+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6dc0558b0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.679+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6ec0b0bd0 con 0x7fc6ec0cff00 2024-03-24T10:34:28.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.680+0000 7fc6e97fa640 1 -- 172.21.15.1:0/1333988124 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fc6dc00ce60 con 0x7fc6ec0cff00 2024-03-24T10:34:28.680 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:28.680 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:28.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 msgr2=0x7fc6cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 0x7fc6cc06dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009a20 tx=0x7fc6e4009190 comp rx=0 tx=0).stop 2024-03-24T10:34:28.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 msgr2=0x7fc6ec0aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00efc0 tx=0x7fc6dc009f90 comp rx=0 tx=0).stop 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 shutdown_connections 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6cc06b810 0x7fc6cc06dcd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ec15bda0 0x7fc6ec1f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc6ec0cff00 0x7fc6ec0aed70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 --2- 172.21.15.1:0/1333988124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec06c910 0x7fc6ec154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 >> 172.21.15.1:0/1333988124 conn(0x7fc6ec0b17a0 msgr2=0x7fc6ec0cc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 shutdown_connections 2024-03-24T10:34:28.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.682+0000 7fc6f2d2c640 1 -- 172.21.15.1:0/1333988124 wait complete. 2024-03-24T10:34:28.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:28.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:28.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.798+0000 7f660c6ed640 1 Processor -- start 2024-03-24T10:34:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.798+0000 7f660c6ed640 1 -- start start 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660c6ed640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66040d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660c6ed640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660c6ed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66040b0b70 con 0x7f66040b1300 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660c6ed640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f66040d1d60 con 0x7f66040b2ed0 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660c6ed640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f66040b0de0 con 0x7f660415e390 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f6609c88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660ac8a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f66040b1300 0x7f66040b1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33108/0 (socket says 172.21.15.1:33108) 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f6609c88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35056/0 (socket says 172.21.15.1:35056) 2024-03-24T10:34:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660ac8a640 1 -- 172.21.15.1:0/1065003364 learned_addr learned my addr 172.21.15.1:0/1065003364 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.799+0000 7f660a489640 1 --2- 172.21.15.1:0/1065003364 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66040d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f660a489640 1 -- 172.21.15.1:0/1065003364 >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 msgr2=0x7f66040d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f660a489640 1 --2- 172.21.15.1:0/1065003364 >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f660a489640 1 -- 172.21.15.1:0/1065003364 >> v1:172.21.15.1:6789/0 conn(0x7f66040b1300 legacy=0x7f66040b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f660a489640 1 -- 172.21.15.1:0/1065003364 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660415e770 con 0x7f660415e390 2024-03-24T10:34:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f6609c88640 1 --2- 172.21.15.1:0/1065003364 >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.800+0000 7f660a489640 1 --2- 172.21.15.1:0/1065003364 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66040d2310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0097a0 tx=0x7f65fc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e791ab7b7adf83de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f6609487640 1 -- 172.21.15.1:0/1065003364 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc031830 con 0x7f660415e390 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f6609487640 1 -- 172.21.15.1:0/1065003364 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65fc031990 con 0x7f660415e390 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f6609487640 1 -- 172.21.15.1:0/1065003364 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc031d20 con 0x7f660415e390 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 -- 172.21.15.1:0/1065003364 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 msgr2=0x7f66040d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 --2- 172.21.15.1:0/1065003364 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66040d2310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0097a0 tx=0x7f65fc02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 -- 172.21.15.1:0/1065003364 shutdown_connections 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 --2- 172.21.15.1:0/1065003364 >> v2:172.21.15.1:3300/0 conn(0x7f66040b2ed0 0x7f66040d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 --2- 172.21.15.1:0/1065003364 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66040d2310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 -- 172.21.15.1:0/1065003364 >> 172.21.15.1:0/1065003364 conn(0x7f66040d3ee0 msgr2=0x7f66040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 -- 172.21.15.1:0/1065003364 shutdown_connections 2024-03-24T10:34:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.801+0000 7f660c6ed640 1 -- 172.21.15.1:0/1065003364 wait complete. 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.802+0000 7f660c6ed640 1 Processor -- start 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 -- start start 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 0x7f660415a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66041605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6604158690 con 0x7f66040b2ed0 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f66041598e0 con 0x7f66040b1300 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660c6ed640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6604159bc0 con 0x7f660415e390 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f6609c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f6609c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35068/0 (socket says 172.21.15.1:35068) 2024-03-24T10:34:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f6609c88640 1 -- 172.21.15.1:0/4029416882 learned_addr learned my addr 172.21.15.1:0/4029416882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660ac8a640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66041605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.803+0000 7f660a489640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 0x7f660415a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 -- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 msgr2=0x7f66041605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66041605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 -- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 msgr2=0x7f660415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 0x7f660415a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 -- 172.21.15.1:0/4029416882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f8009400 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f660a489640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 0x7f660415a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6609c88640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f65f800ebe0 tx=0x7f65f800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8009760 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f800cc30 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc009400 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f800eee0 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.804+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660415a180 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.805+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f660404f4e0 con 0x7f66040b2ed0 2024-03-24T10:34:28.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.805+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f65f8013470 con 0x7f66040b2ed0 2024-03-24T10:34:28.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.806+0000 7f6602ffd640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.806+0000 7f660a489640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.806+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f65f808a4a0 con 0x7f66040b2ed0 2024-03-24T10:34:28.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.806+0000 7f660a489640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65fc036000 tx=0x7f65fc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.810+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f65f8054f80 con 0x7f66040b2ed0 2024-03-24T10:34:28.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.990+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f66040ca840 con 0x7f66040b2ed0 2024-03-24T10:34:28.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.990+0000 7f6602ffd640 1 -- 172.21.15.1:0/4029416882 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f65f8058640 con 0x7f66040b2ed0 2024-03-24T10:34:28.991 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 msgr2=0x7f65f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65fc036000 tx=0x7f65fc03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 msgr2=0x7f660414f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f65f800ebe0 tx=0x7f65f800c520 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 shutdown_connections 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660415e390 0x7f66041605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66040b2ed0 0x7f660414f430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 --2- 172.21.15.1:0/4029416882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66040b1300 0x7f660415a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 >> 172.21.15.1:0/4029416882 conn(0x7f66040d3ee0 msgr2=0x7f66040d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 shutdown_connections 2024-03-24T10:34:28.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:28.992+0000 7f660c6ed640 1 -- 172.21.15.1:0/4029416882 wait complete. 2024-03-24T10:34:29.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.107+0000 7fef35df4640 1 Processor -- start 2024-03-24T10:34:29.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.107+0000 7fef35df4640 1 -- start start 2024-03-24T10:34:29.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef35df4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef3015e860 0x7fef300cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef35df4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef300b45c0 con 0x7fef300d04c0 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef35df4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef300d0920 con 0x7fef300cbf90 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef35df4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fef3006c930 con 0x7fef3015e860 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef3015e860 0x7fef300cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef300d04c0 0x7fef3006cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33120/0 (socket says 172.21.15.1:33120) 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2ffff640 1 -- 172.21.15.1:0/989826686 learned_addr learned my addr 172.21.15.1:0/989826686 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef3015e860 0x7fef300cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35076/0 (socket says 172.21.15.1:35076) 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2effd640 1 -- 172.21.15.1:0/989826686 >> v1:172.21.15.3:6789/0 conn(0x7fef300cbf90 legacy=0x7fef3015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2effd640 1 -- 172.21.15.1:0/989826686 >> v1:172.21.15.1:6789/0 conn(0x7fef300d04c0 legacy=0x7fef3006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2effd640 1 -- 172.21.15.1:0/989826686 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3015ed00 con 0x7fef3015e860 2024-03-24T10:34:29.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.108+0000 7fef2e7fc640 1 -- 172.21.15.1:0/989826686 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269895073 0 0) 0x7fef300b45c0 con 0x7fef300d04c0 2024-03-24T10:34:29.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.109+0000 7fef2effd640 1 --2- 172.21.15.1:0/989826686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef3015e860 0x7fef300cff80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef240097a0 tx=0x7fef2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca72fd5ddce7b753 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.109+0000 7fef2e7fc640 1 -- 172.21.15.1:0/989826686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24031830 con 0x7fef3015e860 2024-03-24T10:34:29.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.109+0000 7fef2e7fc640 1 -- 172.21.15.1:0/989826686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef24031990 con 0x7fef3015e860 2024-03-24T10:34:29.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.109+0000 7fef2e7fc640 1 -- 172.21.15.1:0/989826686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24031d20 con 0x7fef3015e860 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 -- 172.21.15.1:0/989826686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef3015e860 msgr2=0x7fef300cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 --2- 172.21.15.1:0/989826686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef3015e860 0x7fef300cff80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef240097a0 tx=0x7fef2402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 -- 172.21.15.1:0/989826686 shutdown_connections 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 --2- 172.21.15.1:0/989826686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef3015e860 0x7fef300cff80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 -- 172.21.15.1:0/989826686 >> 172.21.15.1:0/989826686 conn(0x7fef3006b4f0 msgr2=0x7fef3006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 -- 172.21.15.1:0/989826686 shutdown_connections 2024-03-24T10:34:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.110+0000 7fef35df4640 1 -- 172.21.15.1:0/989826686 wait complete. 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.111+0000 7fef35df4640 1 Processor -- start 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.111+0000 7fef35df4640 1 -- start start 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef300d04c0 0x7fef30070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef3015e860 0x7fef300c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fef30156980 con 0x7fef300cbf90 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fef30156c60 con 0x7fef300d04c0 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef35df4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fef301f5880 con 0x7fef3015e860 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35094/0 (socket says 172.21.15.1:35094) 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 -- 172.21.15.1:0/3909406475 learned_addr learned my addr 172.21.15.1:0/3909406475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2effd640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef300d04c0 0x7fef30070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2ffff640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef3015e860 0x7fef300c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 -- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef3015e860 msgr2=0x7fef300c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef3015e860 0x7fef300c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 -- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef300d04c0 msgr2=0x7fef30070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef300d04c0 0x7fef30070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 -- 172.21.15.1:0/3909406475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef24009400 con 0x7fef300cbf90 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.112+0000 7fef2f7fe640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c480 tx=0x7fef2000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.113+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef200052d0 con 0x7fef300cbf90 2024-03-24T10:34:29.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.113+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef20005430 con 0x7fef300cbf90 2024-03-24T10:34:29.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.113+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef301f5b50 con 0x7fef300cbf90 2024-03-24T10:34:29.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.113+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20012870 con 0x7fef300cbf90 2024-03-24T10:34:29.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.113+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301f6110 con 0x7fef300cbf90 2024-03-24T10:34:29.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.114+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef301f62a0 con 0x7fef300cbf90 2024-03-24T10:34:29.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.114+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fef200129d0 con 0x7fef300cbf90 2024-03-24T10:34:29.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.114+0000 7fef17fff640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 0x7fef1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.115+0000 7fef2effd640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 0x7fef1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.115+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef20089b90 con 0x7fef300cbf90 2024-03-24T10:34:29.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.115+0000 7fef2effd640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 0x7fef1c06dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef24036000 tx=0x7fef2403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.119+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fef301f62a0 con 0x7fef300cbf90 2024-03-24T10:34:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.300+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef301f62a0 con 0x7fef300cbf90 2024-03-24T10:34:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.301+0000 7fef17fff640 1 -- 172.21.15.1:0/3909406475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fef200546f0 con 0x7fef300cbf90 2024-03-24T10:34:29.301 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:29.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 msgr2=0x7fef1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 0x7fef1c06dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef24036000 tx=0x7fef2403c040 comp rx=0 tx=0).stop 2024-03-24T10:34:29.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 msgr2=0x7fef301f5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c480 tx=0x7fef2000c940 comp rx=0 tx=0).stop 2024-03-24T10:34:29.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 shutdown_connections 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fef1c06b810 0x7fef1c06dcd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fef3015e860 0x7fef300c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef300d04c0 0x7fef30070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 --2- 172.21.15.1:0/3909406475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef300cbf90 0x7fef301f5340 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 >> 172.21.15.1:0/3909406475 conn(0x7fef3006b4f0 msgr2=0x7fef300cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 shutdown_connections 2024-03-24T10:34:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.303+0000 7fef35df4640 1 -- 172.21.15.1:0/3909406475 wait complete. 2024-03-24T10:34:29.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:29.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.418+0000 7fb70dbbc640 1 Processor -- start 2024-03-24T10:34:29.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.418+0000 7fb70dbbc640 1 -- start start 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb70dbbc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb708145730 0x7fb70815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb70dbbc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb70815ff10 0x7fb7080cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb70dbbc640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb708160340 con 0x7fb70806d120 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb70dbbc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb708145b10 con 0x7fb708145730 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb70dbbc640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb70806d620 con 0x7fb70815ff10 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb7077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb708145730 0x7fb70815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb7077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb708145730 0x7fb70815f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35098/0 (socket says 172.21.15.1:35098) 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb7077fe640 1 -- 172.21.15.1:0/3503925830 learned_addr learned my addr 172.21.15.1:0/3503925830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.419+0000 7fb707fff640 1 --2- 172.21.15.1:0/3503925830 >> v2:172.21.15.3:3300/0 conn(0x7fb70815ff10 0x7fb7080cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7077fe640 1 -- 172.21.15.1:0/3503925830 >> v2:172.21.15.3:3300/0 conn(0x7fb70815ff10 msgr2=0x7fb7080cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:29.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3503925830 >> v2:172.21.15.3:3300/0 conn(0x7fb70815ff10 0x7fb7080cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7077fe640 1 -- 172.21.15.1:0/3503925830 >> v1:172.21.15.3:6789/0 conn(0x7fb70806d120 legacy=0x7fb708070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7077fe640 1 -- 172.21.15.1:0/3503925830 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70806d900 con 0x7fb708145730 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3503925830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708145730 0x7fb70815f9d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7080729f0 tx=0x7fb6f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4e4244388bbb42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7067fc640 1 -- 172.21.15.1:0/3503925830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8030860 con 0x7fb708145730 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7067fc640 1 -- 172.21.15.1:0/3503925830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f80309c0 con 0x7fb708145730 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.420+0000 7fb7067fc640 1 -- 172.21.15.1:0/3503925830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8030d50 con 0x7fb708145730 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3503925830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708145730 msgr2=0x7fb70815f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3503925830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708145730 0x7fb70815f9d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7080729f0 tx=0x7fb6f802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3503925830 shutdown_connections 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3503925830 >> v2:172.21.15.3:3300/0 conn(0x7fb70815ff10 0x7fb7080cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3503925830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708145730 0x7fb70815f9d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3503925830 >> 172.21.15.1:0/3503925830 conn(0x7fb7080b1710 msgr2=0x7fb7080b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3503925830 shutdown_connections 2024-03-24T10:34:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.421+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3503925830 wait complete. 2024-03-24T10:34:29.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.422+0000 7fb70dbbc640 1 Processor -- start 2024-03-24T10:34:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.422+0000 7fb70dbbc640 1 -- start start 2024-03-24T10:34:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 0x7fb70815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb708145730 0x7fb7080aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb70815d6b0 con 0x7fb70815ff10 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb70815ca30 con 0x7fb70806d120 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb70dbbc640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb70815cd10 con 0x7fb708145730 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35100/0 (socket says 172.21.15.1:35100) 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 -- 172.21.15.1:0/3130673079 learned_addr learned my addr 172.21.15.1:0/3130673079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 0x7fb70815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb706ffd640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb708145730 0x7fb7080aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 -- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb708145730 msgr2=0x7fb7080aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb708145730 0x7fb7080aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 -- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 msgr2=0x7fb70815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 0x7fb70815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb707fff640 1 -- 172.21.15.1:0/3130673079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f80094d0 con 0x7fb70815ff10 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.423+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 0x7fb70815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb707fff640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb70000efc0 tx=0x7fb70000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700009ba0 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70000f040 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7000146c0 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70815cff0 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.424+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7080af240 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.425+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70815d7f0 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.425+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb700014820 con 0x7fb70815ff10 2024-03-24T10:34:29.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.426+0000 7fb6effff640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 0x7fb6f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.426+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb70008ae60 con 0x7fb70815ff10 2024-03-24T10:34:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.426+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 0x7fb6f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.426+0000 7fb7077fe640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 0x7fb6f406dcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8036000 tx=0x7fb6f8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.430+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb700055930 con 0x7fb70815ff10 2024-03-24T10:34:29.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.610+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb70804f4e0 con 0x7fb70815ff10 2024-03-24T10:34:29.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.610+0000 7fb6effff640 1 -- 172.21.15.1:0/3130673079 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fb7000180a0 con 0x7fb70815ff10 2024-03-24T10:34:29.611 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 msgr2=0x7fb6f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 0x7fb6f406dcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8036000 tx=0x7fb6f8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 msgr2=0x7fb7081f2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb70000efc0 tx=0x7fb70000c490 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 shutdown_connections 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb6f406b810 0x7fb6f406dcd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70815ff10 0x7fb7081f2ae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb708145730 0x7fb7080aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 --2- 172.21.15.1:0/3130673079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb70806d120 0x7fb70815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 >> 172.21.15.1:0/3130673079 conn(0x7fb7080b1710 msgr2=0x7fb70815f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 shutdown_connections 2024-03-24T10:34:29.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.612+0000 7fb70dbbc640 1 -- 172.21.15.1:0/3130673079 wait complete. 2024-03-24T10:34:29.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:29.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.725+0000 7f56e0a64640 1 Processor -- start 2024-03-24T10:34:29.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.725+0000 7f56e0a64640 1 -- start start 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56e0a64640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 0x7f56dc0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56e0a64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0b6a40 0x7f56dc06d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56e0a64640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56dc0b4c70 con 0x7f56dc06cba0 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56e0a64640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f56dc0d1320 con 0x7f56dc0b6a40 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56e0a64640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f56dc06d550 con 0x7f56dc0b7530 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56dad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0b6a40 0x7f56dc06d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56da59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 0x7f56dc0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56dad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56dc0b6a40 0x7f56dc06d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35104/0 (socket says 172.21.15.1:35104) 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.726+0000 7f56dad9d640 1 -- 172.21.15.1:0/1521586305 learned_addr learned my addr 172.21.15.1:0/1521586305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:29.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56d959a640 1 -- 172.21.15.1:0/1521586305 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086169686 0 0) 0x7f56dc0b4c70 con 0x7f56dc06cba0 2024-03-24T10:34:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56dad9d640 1 -- 172.21.15.1:0/1521586305 >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 msgr2=0x7f56dc0d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56dad9d640 1 --2- 172.21.15.1:0/1521586305 >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 0x7f56dc0d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56dad9d640 1 -- 172.21.15.1:0/1521586305 >> v1:172.21.15.3:6789/0 conn(0x7f56dc06cba0 legacy=0x7f56dc161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56dad9d640 1 -- 172.21.15.1:0/1521586305 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc0b4f50 con 0x7f56dc0b6a40 2024-03-24T10:34:29.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56da59c640 1 --2- 172.21.15.1:0/1521586305 >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 0x7f56dc0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56dad9d640 1 --2- 172.21.15.1:0/1521586305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b6a40 0x7f56dc06d010 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f56dc09fda0 tx=0x7f56cc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30deaedc1412c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.727+0000 7f56d959a640 1 -- 172.21.15.1:0/1521586305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc02cb00 con 0x7f56dc0b6a40 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56d959a640 1 -- 172.21.15.1:0/1521586305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56cc02cc60 con 0x7f56dc0b6a40 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56d959a640 1 -- 172.21.15.1:0/1521586305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0343f0 con 0x7f56dc0b6a40 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 -- 172.21.15.1:0/1521586305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b6a40 msgr2=0x7f56dc06d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 --2- 172.21.15.1:0/1521586305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b6a40 0x7f56dc06d010 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f56dc09fda0 tx=0x7f56cc02bf60 comp rx=0 tx=0).stop 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 -- 172.21.15.1:0/1521586305 shutdown_connections 2024-03-24T10:34:29.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 --2- 172.21.15.1:0/1521586305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b6a40 0x7f56dc06d010 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 --2- 172.21.15.1:0/1521586305 >> v2:172.21.15.3:3300/0 conn(0x7f56dc0b7530 0x7f56dc0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 -- 172.21.15.1:0/1521586305 >> 172.21.15.1:0/1521586305 conn(0x7f56dc0d1c80 msgr2=0x7f56dc0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 -- 172.21.15.1:0/1521586305 shutdown_connections 2024-03-24T10:34:29.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.728+0000 7f56e0a64640 1 -- 172.21.15.1:0/1521586305 wait complete. 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.729+0000 7f56e0a64640 1 Processor -- start 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.729+0000 7f56e0a64640 1 -- start start 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f56dc06cba0 0x7f56dc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56dc0b6a40 0x7f56dc0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc0ca8d0 con 0x7f56dc0b7530 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc0cbbc0 con 0x7f56dc0b6a40 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56e0a64640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc0cab20 con 0x7f56dc06cba0 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56dad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35120/0 (socket says 172.21.15.1:35120) 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56dad9d640 1 -- 172.21.15.1:0/3561817645 learned_addr learned my addr 172.21.15.1:0/3561817645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:29.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56da59c640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f56dc06cba0 0x7f56dc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.730+0000 7f56d9d9b640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56dc0b6a40 0x7f56dc0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 -- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f56dc06cba0 msgr2=0x7f56dc15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f56dc06cba0 0x7f56dc15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 -- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56dc0b6a40 msgr2=0x7f56dc0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56dc0b6a40 0x7f56dc0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 -- 172.21.15.1:0/3561817645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d4009400 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56dad9d640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f56cc036040 tx=0x7f56cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc009510 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56cc03e450 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc03e5e0 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56cc009b30 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.731+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56dc0cb0e0 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.732+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56dc06de20 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.732+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56cc03e740 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.733+0000 7f56d2ffd640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 0x7f56bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.733+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56cc0ac9a0 con 0x7f56dc0b7530 2024-03-24T10:34:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.733+0000 7f56da59c640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 0x7f56bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.733+0000 7f56da59c640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 0x7f56bc06dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009c30 tx=0x7f56d4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:29.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.737+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56cc077470 con 0x7f56dc0b7530 2024-03-24T10:34:29.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.933+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f56dc04f4e0 con 0x7f56dc0b7530 2024-03-24T10:34:29.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.933+0000 7f56d2ffd640 1 -- 172.21.15.1:0/3561817645 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f56cc07ab30 con 0x7f56dc0b7530 2024-03-24T10:34:29.933 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 msgr2=0x7f56bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 0x7f56bc06dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009c30 tx=0x7f56d4009190 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 msgr2=0x7f56dc0c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f56cc036040 tx=0x7f56cc004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 shutdown_connections 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f56bc06b810 0x7f56bc06dcd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc0b7530 0x7f56dc0c78f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56dc0b6a40 0x7f56dc0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 --2- 172.21.15.1:0/3561817645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f56dc06cba0 0x7f56dc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 >> 172.21.15.1:0/3561817645 conn(0x7f56dc0d1c80 msgr2=0x7f56dc0ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 shutdown_connections 2024-03-24T10:34:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:29.935+0000 7f56e0a64640 1 -- 172.21.15.1:0/3561817645 wait complete. 2024-03-24T10:34:29.946 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:34:29.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:29.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:34:29.973 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:29.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:34:29.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:29.974 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:34:30.028 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:30.029 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:30.044 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:30.044 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:30.100 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.201+0000 7f9e9d207640 1 Processor -- start 2024-03-24T10:34:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.201+0000 7f9e9d207640 1 -- start start 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e9d207640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e980d1000 0x7f9e980b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e9d207640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9815e770 con 0x7f9e9815e390 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e9d207640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e980d4e40 con 0x7f9e980d2bb0 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e9d207640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9e980b3190 con 0x7f9e980d1000 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e96d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e9815e390 0x7f9e980d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33128/0 (socket says 172.21.15.1:33128) 2024-03-24T10:34:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e96d9d640 1 -- 172.21.15.1:0/3958752757 learned_addr learned my addr 172.21.15.1:0/3958752757 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.202+0000 7f9e9759e640 1 --2- 172.21.15.1:0/3958752757 >> v2:172.21.15.3:3300/0 conn(0x7f9e980d1000 0x7f9e980b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.203+0000 7f9e9659c640 1 -- 172.21.15.1:0/3958752757 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396940627 0 0) 0x7f9e88001110 con 0x7f9e9815e390 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.203+0000 7f9e9659c640 1 -- 172.21.15.1:0/3958752757 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e84003520 con 0x7f9e9815e390 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.203+0000 7f9e9759e640 1 -- 172.21.15.1:0/3958752757 >> v1:172.21.15.3:6789/0 conn(0x7f9e980d2bb0 legacy=0x7f9e980b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.203+0000 7f9e9759e640 1 -- 172.21.15.1:0/3958752757 >> v1:172.21.15.1:6789/0 conn(0x7f9e9815e390 legacy=0x7f9e980d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.203+0000 7f9e9759e640 1 -- 172.21.15.1:0/3958752757 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e980d1e40 con 0x7f9e980d1000 2024-03-24T10:34:30.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9759e640 1 --2- 172.21.15.1:0/3958752757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d1000 0x7f9e980b42a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009d90 tx=0x7f9e9002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75f791f6c94357a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9659c640 1 -- 172.21.15.1:0/3958752757 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9002cbd0 con 0x7f9e980d1000 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9659c640 1 -- 172.21.15.1:0/3958752757 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e9002cd30 con 0x7f9e980d1000 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9659c640 1 -- 172.21.15.1:0/3958752757 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9002cea0 con 0x7f9e980d1000 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 -- 172.21.15.1:0/3958752757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d1000 msgr2=0x7f9e980b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 --2- 172.21.15.1:0/3958752757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d1000 0x7f9e980b42a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009d90 tx=0x7f9e9002be20 comp rx=0 tx=0).stop 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 -- 172.21.15.1:0/3958752757 shutdown_connections 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 --2- 172.21.15.1:0/3958752757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d1000 0x7f9e980b42a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 -- 172.21.15.1:0/3958752757 >> 172.21.15.1:0/3958752757 conn(0x7f9e980d3ec0 msgr2=0x7f9e980d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 -- 172.21.15.1:0/3958752757 shutdown_connections 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.204+0000 7f9e9d207640 1 -- 172.21.15.1:0/3958752757 wait complete. 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 Processor -- start 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 -- start start 2024-03-24T10:34:30.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d2bb0 0x7f9e980cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9e9815e390 0x7f9e981517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9e98152520 con 0x7f9e980d1000 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9e98152800 con 0x7f9e980d2bb0 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.206+0000 7f9e9d207640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9e98152ae0 con 0x7f9e9815e390 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35124/0 (socket says 172.21.15.1:35124) 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 -- 172.21.15.1:0/379342493 learned_addr learned my addr 172.21.15.1:0/379342493 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:30.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e9759e640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9e9815e390 0x7f9e981517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e8ffff640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d2bb0 0x7f9e980cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 -- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9e9815e390 msgr2=0x7f9e981517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9e9815e390 0x7f9e981517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 -- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d2bb0 msgr2=0x7f9e980cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d2bb0 0x7f9e980cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 -- 172.21.15.1:0/379342493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e90009a70 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e96d9d640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c420 tx=0x7f9e8800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88005ce0 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e88005e40 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9809dc10 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e880072f0 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.207+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e980b98d0 con 0x7f9e980d1000 2024-03-24T10:34:30.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.208+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9804f4e0 con 0x7f9e980d1000 2024-03-24T10:34:30.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.209+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9e88007450 con 0x7f9e980d1000 2024-03-24T10:34:30.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.209+0000 7f9e8f7fe640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 0x7f9e7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.209+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9e880895c0 con 0x7f9e980d1000 2024-03-24T10:34:30.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.209+0000 7f9e8ffff640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 0x7f9e7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.209+0000 7f9e8ffff640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 0x7f9e7806dd00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e980b7af0 tx=0x7f9e80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.213+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9e88054090 con 0x7f9e980d1000 2024-03-24T10:34:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.397+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e9815e770 con 0x7f9e980d1000 2024-03-24T10:34:30.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.398+0000 7f9e8f7fe640 1 -- 172.21.15.1:0/379342493 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7f9e9815e770 con 0x7f9e980d1000 2024-03-24T10:34:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:30.398 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:30.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 msgr2=0x7f9e7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 0x7f9e7806dd00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e980b7af0 tx=0x7f9e80009040 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 msgr2=0x7f9e980cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c420 tx=0x7f9e8800c8e0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 shutdown_connections 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9e7806b840 0x7f9e7806dd00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9e9815e390 0x7f9e981517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e980d2bb0 0x7f9e980cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 --2- 172.21.15.1:0/379342493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e980d1000 0x7f9e980cd760 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 >> 172.21.15.1:0/379342493 conn(0x7f9e980d3ec0 msgr2=0x7f9e980b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 shutdown_connections 2024-03-24T10:34:30.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.400+0000 7f9e9d207640 1 -- 172.21.15.1:0/379342493 wait complete. 2024-03-24T10:34:30.411 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:30.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 Processor -- start 2024-03-24T10:34:30.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 -- start start 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7028154be0 0x7f702806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70280cf450 0x7f70280b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f70280b40c0 0x7f702815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7028070bd0 con 0x7f70280cf450 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f7028154fc0 con 0x7f70280b40c0 2024-03-24T10:34:30.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f702ecb5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7028071370 con 0x7f7028154be0 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.516+0000 7f7027fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70280cf450 0x7f70280b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70280cf450 0x7f70280b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35136/0 (socket says 172.21.15.1:35136) 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 -- 172.21.15.1:0/2974786252 learned_addr learned my addr 172.21.15.1:0/2974786252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f702d252640 1 --2- 172.21.15.1:0/2974786252 >> v2:172.21.15.3:3300/0 conn(0x7f70280b40c0 0x7f702815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f702ca51640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7028154be0 0x7f702806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 -- 172.21.15.1:0/2974786252 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7028154be0 msgr2=0x7f702806d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7028154be0 0x7f702806d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 -- 172.21.15.1:0/2974786252 >> v2:172.21.15.3:3300/0 conn(0x7f70280b40c0 msgr2=0x7f702815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 --2- 172.21.15.1:0/2974786252 >> v2:172.21.15.3:3300/0 conn(0x7f70280b40c0 0x7f702815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.517+0000 7f7027fff640 1 -- 172.21.15.1:0/2974786252 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70280b36a0 con 0x7f70280cf450 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f7027fff640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70280cf450 0x7f70280b3b80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f701400aa10 tx=0x7f701402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de5f0594e76ad49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f70277fe640 1 -- 172.21.15.1:0/2974786252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701402dae0 con 0x7f70280cf450 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f70277fe640 1 -- 172.21.15.1:0/2974786252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701402dc40 con 0x7f70280cf450 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f70277fe640 1 -- 172.21.15.1:0/2974786252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014035440 con 0x7f70280cf450 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 -- 172.21.15.1:0/2974786252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70280cf450 msgr2=0x7f70280b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70280cf450 0x7f70280b3b80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f701400aa10 tx=0x7f701402cd30 comp rx=0 tx=0).stop 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 -- 172.21.15.1:0/2974786252 shutdown_connections 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 --2- 172.21.15.1:0/2974786252 >> v2:172.21.15.3:3300/0 conn(0x7f70280b40c0 0x7f702815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70280cf450 0x7f70280b3b80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 --2- 172.21.15.1:0/2974786252 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7028154be0 0x7f702806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 -- 172.21.15.1:0/2974786252 >> 172.21.15.1:0/2974786252 conn(0x7f70280b1710 msgr2=0x7f70280b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 -- 172.21.15.1:0/2974786252 shutdown_connections 2024-03-24T10:34:30.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.518+0000 7f702ecb5640 1 -- 172.21.15.1:0/2974786252 wait complete. 2024-03-24T10:34:30.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.519+0000 7f702ecb5640 1 Processor -- start 2024-03-24T10:34:30.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.519+0000 7f702ecb5640 1 -- start start 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70280b40c0 0x7f702814a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 0x7f70280cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f70281e3d30 con 0x7f7028154be0 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f70281e3a50 con 0x7f70280b40c0 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702ecb5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f702814acb0 con 0x7f70280cf450 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702d252640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702d252640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35146/0 (socket says 172.21.15.1:35146) 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.520+0000 7f702d252640 1 -- 172.21.15.1:0/1093716534 learned_addr learned my addr 172.21.15.1:0/1093716534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:30.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 0x7f70280cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f702ca51640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70280b40c0 0x7f702814a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 -- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70280b40c0 msgr2=0x7f702814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70280b40c0 0x7f702814a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 -- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 msgr2=0x7f70280cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 -- 172.21.15.1:0/1093716534 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701400a670 con 0x7f70280cf450 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f702d252640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f7027fff640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 0x7f70280cd890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7028073890 tx=0x7f70140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701403f600 con 0x7f70280cf450 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701403f780 con 0x7f70280cf450 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702814af90 con 0x7f70280cf450 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701403fb30 con 0x7f70280cf450 2024-03-24T10:34:30.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.521+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f702814a540 con 0x7f70280cf450 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.522+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f702814ab10 con 0x7f70280cf450 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.523+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f701403fc90 con 0x7f70280cf450 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.523+0000 7f70257fa640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.523+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f70140ac4a0 con 0x7f70280cf450 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.523+0000 7f702ca51640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.524+0000 7f702ca51640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 0x7f700c06dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f701c009b40 tx=0x7f701c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.528+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f702814ab10 con 0x7f70280cf450 2024-03-24T10:34:30.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.708+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f702814ab10 con 0x7f70280cf450 2024-03-24T10:34:30.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.708+0000 7f70257fa640 1 -- 172.21.15.1:0/1093716534 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f702814ab10 con 0x7f70280cf450 2024-03-24T10:34:30.712 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:30.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 msgr2=0x7f700c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 0x7f700c06dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f701c009b40 tx=0x7f701c009190 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 msgr2=0x7f70280cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 0x7f70280cd890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7028073890 tx=0x7f70140046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 shutdown_connections 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7028154be0 0x7f70280cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70280cf450 0x7f70280cd890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 --2- 172.21.15.1:0/1093716534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70280b40c0 0x7f702814a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 >> 172.21.15.1:0/1093716534 conn(0x7f70280b1710 msgr2=0x7f702806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 shutdown_connections 2024-03-24T10:34:30.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.710+0000 7f702ecb5640 1 -- 172.21.15.1:0/1093716534 wait complete. 2024-03-24T10:34:30.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.825+0000 7fd0e98f4640 1 Processor -- start 2024-03-24T10:34:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.825+0000 7fd0e98f4640 1 -- start start 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e98f4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0e40b8d20 0x7fd0e4168c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e98f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e40cfc70 con 0x7fd0e40d2540 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e98f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e40b6f50 con 0x7fd0e40d1e20 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e98f4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e40cfeb0 con 0x7fd0e40b8d20 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e2ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd0e40d2540 0x7fd0e40d2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52760/0 (socket says 172.21.15.1:52760) 2024-03-24T10:34:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e2ffd640 1 -- 172.21.15.1:0/4187660488 learned_addr learned my addr 172.21.15.1:0/4187660488 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e37fe640 1 --2- 172.21.15.1:0/4187660488 >> v2:172.21.15.3:3300/0 conn(0x7fd0e40b8d20 0x7fd0e4168c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904581351 0 0) 0x7fd0dc001110 con 0x7fd0e40d2540 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0cc003520 con 0x7fd0e40d2540 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e37fe640 1 -- 172.21.15.1:0/4187660488 >> v1:172.21.15.3:6789/0 conn(0x7fd0e40d1e20 legacy=0x7fd0e40cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246206443 0 0) 0x7fd0e40b6f50 con 0x7fd0e40d1e20 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e37fe640 1 -- 172.21.15.1:0/4187660488 >> v1:172.21.15.1:6789/0 conn(0x7fd0e40d2540 legacy=0x7fd0e40d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.826+0000 7fd0e37fe640 1 -- 172.21.15.1:0/4187660488 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e40b7230 con 0x7fd0e40b8d20 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.827+0000 7fd0e37fe640 1 --2- 172.21.15.1:0/4187660488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40b8d20 0x7fd0e4168c30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009d90 tx=0x7fd0d402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30784d0812321154 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.827+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d402cd80 con 0x7fd0e40b8d20 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.827+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d4034440 con 0x7fd0e40b8d20 2024-03-24T10:34:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.827+0000 7fd0e1ffb640 1 -- 172.21.15.1:0/4187660488 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40347d0 con 0x7fd0e40b8d20 2024-03-24T10:34:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 -- 172.21.15.1:0/4187660488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40b8d20 msgr2=0x7fd0e4168c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/4187660488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40b8d20 0x7fd0e4168c30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009d90 tx=0x7fd0d402bfd0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 -- 172.21.15.1:0/4187660488 shutdown_connections 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/4187660488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40b8d20 0x7fd0e4168c30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 -- 172.21.15.1:0/4187660488 >> 172.21.15.1:0/4187660488 conn(0x7fd0e409d1d0 msgr2=0x7fd0e40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 -- 172.21.15.1:0/4187660488 shutdown_connections 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.828+0000 7fd0e98f4640 1 -- 172.21.15.1:0/4187660488 wait complete. 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.829+0000 7fd0e98f4640 1 Processor -- start 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.829+0000 7fd0e98f4640 1 -- start start 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0e40b8d20 0x7fd0e41d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40d2540 0x7fd0e40cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e40b4910 con 0x7fd0e40d1e20 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e40b3870 con 0x7fd0e40d2540 2024-03-24T10:34:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e98f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e40b3b50 con 0x7fd0e40b8d20 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43304/0 (socket says 172.21.15.1:43304) 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 -- 172.21.15.1:0/2697730779 learned_addr learned my addr 172.21.15.1:0/2697730779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e2ffd640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0e40b8d20 0x7fd0e41d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e37fe640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40d2540 0x7fd0e40cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 -- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0e40b8d20 msgr2=0x7fd0e41d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0e40b8d20 0x7fd0e41d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 -- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40d2540 msgr2=0x7fd0e40cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40d2540 0x7fd0e40cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 -- 172.21.15.1:0/2697730779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d4009a70 con 0x7fd0e40d1e20 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0e27fc640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c4b0 tx=0x7fd0d800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.830+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8005480 con 0x7fd0e40d1e20 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.831+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d80035f0 con 0x7fd0e40d1e20 2024-03-24T10:34:30.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.831+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e40b3e30 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.831+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80039e0 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.831+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e40b43f0 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.831+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e40b45e0 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.832+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0d8003b40 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.832+0000 7fd0cbfff640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 0x7fd0d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.832+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd0d8089710 con 0x7fd0e40d1e20 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.832+0000 7fd0e2ffd640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 0x7fd0d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.833+0000 7fd0e2ffd640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 0x7fd0d006dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc005880 tx=0x7fd0dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:30.837+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0d80541d0 con 0x7fd0e40d1e20 2024-03-24T10:34:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.015+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd0e404f470 con 0x7fd0e40d1e20 2024-03-24T10:34:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.016+0000 7fd0cbfff640 1 -- 172.21.15.1:0/2697730779 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fd0d8057890 con 0x7fd0e40d1e20 2024-03-24T10:34:31.016 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 msgr2=0x7fd0d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 0x7fd0d006dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc005880 tx=0x7fd0dc009040 comp rx=0 tx=0).stop 2024-03-24T10:34:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 msgr2=0x7fd0e40b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c4b0 tx=0x7fd0d800c970 comp rx=0 tx=0).stop 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 shutdown_connections 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd0d006b810 0x7fd0d006dcd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0e40d2540 0x7fd0e40cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e40d1e20 0x7fd0e40b2e00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 --2- 172.21.15.1:0/2697730779 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd0e40b8d20 0x7fd0e41d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 >> 172.21.15.1:0/2697730779 conn(0x7fd0e409d1d0 msgr2=0x7fd0e40cea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 shutdown_connections 2024-03-24T10:34:31.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.018+0000 7fd0e98f4640 1 -- 172.21.15.1:0/2697730779 wait complete. 2024-03-24T10:34:31.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.133+0000 7f34ccf1e640 1 Processor -- start 2024-03-24T10:34:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.133+0000 7f34ccf1e640 1 -- start start 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34ccf1e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 0x7f34c80cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34ccf1e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c80cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34ccf1e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34c80b0b70 con 0x7f34c815e390 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34ccf1e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f34c80d1d60 con 0x7f34c80d2a20 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34ccf1e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f34c80b0de0 con 0x7f34c80b2310 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34c5d9b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 0x7f34c80cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34c6d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c80cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34c659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f34c815e390 0x7f34c80d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:33982/0 (socket says 172.21.15.1:33982) 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.134+0000 7f34c659c640 1 -- 172.21.15.1:0/732000999 learned_addr learned my addr 172.21.15.1:0/732000999 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c6d9d640 1 -- 172.21.15.1:0/732000999 >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 msgr2=0x7f34c80cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c6d9d640 1 --2- 172.21.15.1:0/732000999 >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 0x7f34c80cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c559a640 1 -- 172.21.15.1:0/732000999 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182253299 0 0) 0x7f34c80b0b70 con 0x7f34c815e390 2024-03-24T10:34:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c6d9d640 1 -- 172.21.15.1:0/732000999 >> v1:172.21.15.3:6789/0 conn(0x7f34c815e390 legacy=0x7f34c80d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c6d9d640 1 -- 172.21.15.1:0/732000999 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c815e770 con 0x7f34c80b2310 2024-03-24T10:34:31.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c5d9b640 1 --2- 172.21.15.1:0/732000999 >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 0x7f34c80cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.135+0000 7f34c6d9d640 1 --2- 172.21.15.1:0/732000999 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c80cf010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009d90 tx=0x7f34c002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=797ee2a50ca9b790 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34c559a640 1 -- 172.21.15.1:0/732000999 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c002cc50 con 0x7f34c80b2310 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34c559a640 1 -- 172.21.15.1:0/732000999 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c002cdb0 con 0x7f34c80b2310 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34c559a640 1 -- 172.21.15.1:0/732000999 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0034660 con 0x7f34c80b2310 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 -- 172.21.15.1:0/732000999 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 msgr2=0x7f34c80cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/732000999 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c80cf010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009d90 tx=0x7f34c002bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 -- 172.21.15.1:0/732000999 shutdown_connections 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/732000999 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c80cf010 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/732000999 >> v2:172.21.15.3:3300/0 conn(0x7f34c80d2a20 0x7f34c80cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 -- 172.21.15.1:0/732000999 >> 172.21.15.1:0/732000999 conn(0x7f34c80d3ee0 msgr2=0x7f34c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 -- 172.21.15.1:0/732000999 shutdown_connections 2024-03-24T10:34:31.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.136+0000 7f34ccf1e640 1 -- 172.21.15.1:0/732000999 wait complete. 2024-03-24T10:34:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.137+0000 7f34ccf1e640 1 Processor -- start 2024-03-24T10:34:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 -- start start 2024-03-24T10:34:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c814f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34c815e390 0x7f34c815f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f34c80cfda0 con 0x7f34c80d2a20 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f34c80d0080 con 0x7f34c815e390 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34ccf1e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f34c80d0360 con 0x7f34c80b2310 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34c5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34c5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43320/0 (socket says 172.21.15.1:43320) 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34c6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34c815e390 0x7f34c815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34c5d9b640 1 -- 172.21.15.1:0/905972531 learned_addr learned my addr 172.21.15.1:0/905972531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.138+0000 7f34c659c640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c814f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 -- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 msgr2=0x7f34c814f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c814f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 -- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34c815e390 msgr2=0x7f34c815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34c815e390 0x7f34c815f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 -- 172.21.15.1:0/905972531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b0009400 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34c5d9b640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f34b000ebe0 tx=0x7f34b000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0009760 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b000cc30 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c0009a70 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b000eee0 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.139+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c80b7a80 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.140+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c804f4e0 con 0x7f34c80d2a20 2024-03-24T10:34:31.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.140+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f34b0013470 con 0x7f34c80d2a20 2024-03-24T10:34:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.141+0000 7f34beffd640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 0x7f34a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.141+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f34b008a370 con 0x7f34c80d2a20 2024-03-24T10:34:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.141+0000 7f34c659c640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 0x7f34a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.141+0000 7f34c659c640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 0x7f34a806dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f34b8007fd0 tx=0x7f34b8007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.145+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f34b0054ed0 con 0x7f34c80d2a20 2024-03-24T10:34:31.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.326+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f34c80d3210 con 0x7f34c80d2a20 2024-03-24T10:34:31.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.327+0000 7f34beffd640 1 -- 172.21.15.1:0/905972531 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f34b0058590 con 0x7f34c80d2a20 2024-03-24T10:34:31.327 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 msgr2=0x7f34a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 0x7f34a806dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f34b8007fd0 tx=0x7f34b8007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 msgr2=0x7f34c80cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f34b000ebe0 tx=0x7f34b000c520 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 shutdown_connections 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f34a806b810 0x7f34a806dcd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34c815e390 0x7f34c815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c80d2a20 0x7f34c80cf470 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 --2- 172.21.15.1:0/905972531 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34c80b2310 0x7f34c814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 >> 172.21.15.1:0/905972531 conn(0x7f34c80d3ee0 msgr2=0x7f34c806d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 shutdown_connections 2024-03-24T10:34:31.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.329+0000 7f34ccf1e640 1 -- 172.21.15.1:0/905972531 wait complete. 2024-03-24T10:34:31.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:31.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 Processor -- start 2024-03-24T10:34:31.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 -- start start 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd87c1445e0 0x7fd87c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd87c0cf630 0x7fd87c072010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87c0ce170 con 0x7fd87c0b5120 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd87c1449c0 con 0x7fd87c0cf630 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.447+0000 7fd8819a2640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd87c06f040 con 0x7fd87c1445e0 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd87c0cf630 0x7fd87c072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd87c0cf630 0x7fd87c072010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43330/0 (socket says 172.21.15.1:43330) 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 -- 172.21.15.1:0/4246271067 learned_addr learned my addr 172.21.15.1:0/4246271067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8809a0640 1 --2- 172.21.15.1:0/4246271067 >> v2:172.21.15.3:3300/0 conn(0x7fd87c1445e0 0x7fd87c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 -- 172.21.15.1:0/4246271067 >> v2:172.21.15.3:3300/0 conn(0x7fd87c1445e0 msgr2=0x7fd87c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 --2- 172.21.15.1:0/4246271067 >> v2:172.21.15.3:3300/0 conn(0x7fd87c1445e0 0x7fd87c071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 -- 172.21.15.1:0/4246271067 >> v1:172.21.15.3:6789/0 conn(0x7fd87c0b5120 legacy=0x7fd87c06db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.448+0000 7fd8811a1640 1 -- 172.21.15.1:0/4246271067 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87c06f320 con 0x7fd87c0cf630 2024-03-24T10:34:31.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8811a1640 1 --2- 172.21.15.1:0/4246271067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c0cf630 0x7fd87c072010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd87c068580 tx=0x7fd86402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52731215378595e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd87b7fe640 1 -- 172.21.15.1:0/4246271067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86402f850 con 0x7fd87c0cf630 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd87b7fe640 1 -- 172.21.15.1:0/4246271067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86402f9b0 con 0x7fd87c0cf630 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd87b7fe640 1 -- 172.21.15.1:0/4246271067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86402fd40 con 0x7fd87c0cf630 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 -- 172.21.15.1:0/4246271067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c0cf630 msgr2=0x7fd87c072010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 --2- 172.21.15.1:0/4246271067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c0cf630 0x7fd87c072010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd87c068580 tx=0x7fd86402ba30 comp rx=0 tx=0).stop 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 -- 172.21.15.1:0/4246271067 shutdown_connections 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 --2- 172.21.15.1:0/4246271067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c0cf630 0x7fd87c072010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 --2- 172.21.15.1:0/4246271067 >> v2:172.21.15.3:3300/0 conn(0x7fd87c1445e0 0x7fd87c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 -- 172.21.15.1:0/4246271067 >> 172.21.15.1:0/4246271067 conn(0x7fd87c0b26b0 msgr2=0x7fd87c0b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 -- 172.21.15.1:0/4246271067 shutdown_connections 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.449+0000 7fd8819a2640 1 -- 172.21.15.1:0/4246271067 wait complete. 2024-03-24T10:34:31.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.450+0000 7fd8819a2640 1 Processor -- start 2024-03-24T10:34:31.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 -- start start 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd87c0b5120 0x7fd87c0c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd87c0cf630 0x7fd87c06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c0ad520 con 0x7fd87c1445e0 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c0ad7c0 con 0x7fd87c0b5120 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.451+0000 7fd8819a2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd87c0adaa0 con 0x7fd87c0cf630 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43342/0 (socket says 172.21.15.1:43342) 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8809a0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd87c0b5120 0x7fd87c0c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 -- 172.21.15.1:0/802949920 learned_addr learned my addr 172.21.15.1:0/802949920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd87bfff640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd87c0cf630 0x7fd87c06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 -- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd87c0cf630 msgr2=0x7fd87c06e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd87c0cf630 0x7fd87c06e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 -- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd87c0b5120 msgr2=0x7fd87c0c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd87c0b5120 0x7fd87c0c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 -- 172.21.15.1:0/802949920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8640094d0 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8811a1640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd86402ba70 tx=0x7fd864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864037c40 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd864037dc0 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87c0add80 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864041640 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.452+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87c0abae0 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.453+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd87c0cde90 con 0x7fd87c1445e0 2024-03-24T10:34:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.454+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd8640417a0 con 0x7fd87c1445e0 2024-03-24T10:34:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.454+0000 7fd8797fa640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.454+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd86403c070 con 0x7fd87c1445e0 2024-03-24T10:34:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.454+0000 7fd8809a0640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.455+0000 7fd8809a0640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 0x7fd85c06dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd874009aa0 tx=0x7fd874009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.458+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd8640778f0 con 0x7fd87c1445e0 2024-03-24T10:34:31.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.650+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd87c0adf70 con 0x7fd87c1445e0 2024-03-24T10:34:31.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.651+0000 7fd8797fa640 1 -- 172.21.15.1:0/802949920 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fd86407afb0 con 0x7fd87c1445e0 2024-03-24T10:34:31.651 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 msgr2=0x7fd85c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 0x7fd85c06dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd874009aa0 tx=0x7fd874009190 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 msgr2=0x7fd87c06e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd86402ba70 tx=0x7fd864004730 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 shutdown_connections 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd87c1445e0 0x7fd87c06e6f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd87c0cf630 0x7fd87c06e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.652+0000 7fd8819a2640 1 --2- 172.21.15.1:0/802949920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd87c0b5120 0x7fd87c0c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.653+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 >> 172.21.15.1:0/802949920 conn(0x7fd87c0b26b0 msgr2=0x7fd87c0b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.653+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 shutdown_connections 2024-03-24T10:34:31.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.653+0000 7fd8819a2640 1 -- 172.21.15.1:0/802949920 wait complete. 2024-03-24T10:34:31.664 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:31.664 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:31.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.784+0000 7fa116a1c640 1 Processor -- start 2024-03-24T10:34:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.784+0000 7fa116a1c640 1 -- start start 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa116a1c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 0x7fa11006b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa116a1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1100ce9b0 con 0x7fa11006b5e0 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa116a1c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa11006e540 con 0x7fa11006f390 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa116a1c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa11006e820 con 0x7fa110152d90 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa10f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa11006b5e0 0x7fa11006dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52774/0 (socket says 172.21.15.1:52774) 2024-03-24T10:34:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa10f7fe640 1 -- 172.21.15.1:0/2974295436 learned_addr learned my addr 172.21.15.1:0/2974295436 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa10ffff640 1 --2- 172.21.15.1:0/2974295436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 0x7fa11006b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151438236 0 0) 0x7fa1100ce9b0 con 0x7fa11006b5e0 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.785+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f8003520 con 0x7fa11006b5e0 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084639225 0 0) 0x7fa11006e540 con 0x7fa11006f390 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1100ce9b0 con 0x7fa11006f390 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10ffff640 1 -- 172.21.15.1:0/2974295436 >> v1:172.21.15.3:6789/0 conn(0x7fa11006f390 legacy=0x7fa1100cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10ffff640 1 -- 172.21.15.1:0/2974295436 >> v1:172.21.15.1:6789/0 conn(0x7fa11006b5e0 legacy=0x7fa11006dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10ffff640 1 -- 172.21.15.1:0/2974295436 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1100cec10 con 0x7fa110152d90 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.786+0000 7fa10ffff640 1 --2- 172.21.15.1:0/2974295436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 0x7fa11006b0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097d0 tx=0x7fa10402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b01d3cb9de9aedc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104031890 con 0x7fa110152d90 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1040319f0 con 0x7fa110152d90 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa10effd640 1 -- 172.21.15.1:0/2974295436 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104031d80 con 0x7fa110152d90 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 -- 172.21.15.1:0/2974295436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 msgr2=0x7fa11006b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2974295436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 0x7fa11006b0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097d0 tx=0x7fa10402b930 comp rx=0 tx=0).stop 2024-03-24T10:34:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 -- 172.21.15.1:0/2974295436 shutdown_connections 2024-03-24T10:34:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2974295436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa110152d90 0x7fa11006b0a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 -- 172.21.15.1:0/2974295436 >> 172.21.15.1:0/2974295436 conn(0x7fa1100d0570 msgr2=0x7fa1100d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 -- 172.21.15.1:0/2974295436 shutdown_connections 2024-03-24T10:34:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.787+0000 7fa116a1c640 1 -- 172.21.15.1:0/2974295436 wait complete. 2024-03-24T10:34:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.788+0000 7fa116a1c640 1 Processor -- start 2024-03-24T10:34:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.788+0000 7fa116a1c640 1 -- start start 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa11006f390 0x7fa11014f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa110152d90 0x7fa11014bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa11014db80 con 0x7fa11006b5e0 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa11014fc40 con 0x7fa110152d90 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa116a1c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa11014c0d0 con 0x7fa11006f390 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa10ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43348/0 (socket says 172.21.15.1:43348) 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa10ffff640 1 -- 172.21.15.1:0/2003085946 learned_addr learned my addr 172.21.15.1:0/2003085946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa10f7fe640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa11006f390 0x7fa11014f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.789+0000 7fa114fb9640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa110152d90 0x7fa11014bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 -- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa11006f390 msgr2=0x7fa11014f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa11006f390 0x7fa11014f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 -- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa110152d90 msgr2=0x7fa11014bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa110152d90 0x7fa11014bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 -- 172.21.15.1:0/2003085946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa104009400 con 0x7fa11006b5e0 2024-03-24T10:34:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10ffff640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa104034040 tx=0x7fa1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10403abe0 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10403ad80 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11014cf00 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104044630 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.790+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11014d4c0 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.791+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11004f4e0 con 0x7fa11006b5e0 2024-03-24T10:34:31.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.791+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa10404d450 con 0x7fa11006b5e0 2024-03-24T10:34:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.792+0000 7fa10cff9640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 0x7fa0fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.792+0000 7fa10f7fe640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 0x7fa0fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.792+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1040ac510 con 0x7fa11006b5e0 2024-03-24T10:34:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.792+0000 7fa10f7fe640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 0x7fa0fc06dd00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa11014ebd0 tx=0x7fa100009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:31.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.796+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa104077070 con 0x7fa11006b5e0 2024-03-24T10:34:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.979+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa110072090 con 0x7fa11006b5e0 2024-03-24T10:34:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.980+0000 7fa10cff9640 1 -- 172.21.15.1:0/2003085946 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fa1040aae60 con 0x7fa11006b5e0 2024-03-24T10:34:31.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:31.981 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:31.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 msgr2=0x7fa0fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 0x7fa0fc06dd00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa11014ebd0 tx=0x7fa100009040 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 msgr2=0x7fa11014c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa104034040 tx=0x7fa1040046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 shutdown_connections 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0fc06b840 0x7fa0fc06dd00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa110152d90 0x7fa11014bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa11006f390 0x7fa11014f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 --2- 172.21.15.1:0/2003085946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa11006b5e0 0x7fa11014c890 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 >> 172.21.15.1:0/2003085946 conn(0x7fa1100d0570 msgr2=0x7fa1100b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 shutdown_connections 2024-03-24T10:34:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:31.982+0000 7fa116a1c640 1 -- 172.21.15.1:0/2003085946 wait complete. 2024-03-24T10:34:31.994 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:31.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.098+0000 7fd710ba0640 1 Processor -- start 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.098+0000 7fd710ba0640 1 -- start start 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd710ba0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd70c15e200 0x7fd70c0cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd710ba0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd70c0b35d0 con 0x7fd70c0d0e30 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd710ba0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd70c0b3820 con 0x7fd70c0cf820 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd710ba0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd70c0cfc80 con 0x7fd70c15e200 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd70b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd70c15e200 0x7fd70c0cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd70b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd70c15e200 0x7fd70c0cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43358/0 (socket says 172.21.15.1:43358) 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd70b7fe640 1 -- 172.21.15.1:0/2587349119 learned_addr learned my addr 172.21.15.1:0/2587349119 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.099+0000 7fd70bfff640 1 --1- 172.21.15.1:0/2587349119 >> v1:172.21.15.1:6789/0 conn(0x7fd70c0d0e30 0x7fd70c0b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52790/0 (socket says 172.21.15.1:52790) 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.100+0000 7fd70b7fe640 1 -- 172.21.15.1:0/2587349119 >> v1:172.21.15.3:6789/0 conn(0x7fd70c0cf820 legacy=0x7fd70c0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.100+0000 7fd70b7fe640 1 -- 172.21.15.1:0/2587349119 >> v1:172.21.15.1:6789/0 conn(0x7fd70c0d0e30 legacy=0x7fd70c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.100+0000 7fd70b7fe640 1 -- 172.21.15.1:0/2587349119 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c0b3b00 con 0x7fd70c15e200 2024-03-24T10:34:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.100+0000 7fd70a7fc640 1 -- 172.21.15.1:0/2587349119 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1944733028 0 0) 0x7fd70c0b35d0 con 0x7fd70c0d0e30 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.100+0000 7fd70b7fe640 1 --2- 172.21.15.1:0/2587349119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c0cf2e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009870 tx=0x7fd6fc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159eb6cef76d1caa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd70a7fc640 1 -- 172.21.15.1:0/2587349119 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc02f840 con 0x7fd70c15e200 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd70a7fc640 1 -- 172.21.15.1:0/2587349119 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6fc02f9a0 con 0x7fd70c15e200 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd70a7fc640 1 -- 172.21.15.1:0/2587349119 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc02fd30 con 0x7fd70c15e200 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 -- 172.21.15.1:0/2587349119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 msgr2=0x7fd70c0cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 --2- 172.21.15.1:0/2587349119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c0cf2e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009870 tx=0x7fd6fc02ba20 comp rx=0 tx=0).stop 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 -- 172.21.15.1:0/2587349119 shutdown_connections 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 --2- 172.21.15.1:0/2587349119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c0cf2e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 -- 172.21.15.1:0/2587349119 >> 172.21.15.1:0/2587349119 conn(0x7fd70c0d3d30 msgr2=0x7fd70c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 -- 172.21.15.1:0/2587349119 shutdown_connections 2024-03-24T10:34:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.101+0000 7fd710ba0640 1 -- 172.21.15.1:0/2587349119 wait complete. 2024-03-24T10:34:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.102+0000 7fd710ba0640 1 Processor -- start 2024-03-24T10:34:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.102+0000 7fd710ba0640 1 -- start start 2024-03-24T10:34:32.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd70c0cf820 0x7fd70c1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd70c0d0e30 0x7fd70c0a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c09fc10 con 0x7fd70c15e200 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c09fef0 con 0x7fd70c0cf820 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd710ba0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd70c0a01d0 con 0x7fd70c0d0e30 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd70bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd70bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43366/0 (socket says 172.21.15.1:43366) 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd70bfff640 1 -- 172.21.15.1:0/3559710059 learned_addr learned my addr 172.21.15.1:0/3559710059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd70b7fe640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd70c0cf820 0x7fd70c1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.103+0000 7fd70affd640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd70c0d0e30 0x7fd70c0a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 -- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd70c0d0e30 msgr2=0x7fd70c0a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd70c0d0e30 0x7fd70c0a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 -- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd70c0cf820 msgr2=0x7fd70c1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd70c0cf820 0x7fd70c1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 -- 172.21.15.1:0/3559710059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc0094d0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd70bfff640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c700 tx=0x7fd70400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70400ce10 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70400a640 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c0a04b0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70400aa70 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.104+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1536e0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.105+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd70c0a09d0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.105+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd70400abd0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.106+0000 7fd6f3fff640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 0x7fd6f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.106+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7040898d0 con 0x7fd70c15e200 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.106+0000 7fd70b7fe640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 0x7fd6f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.106+0000 7fd70b7fe640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 0x7fd6f806dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc036000 tx=0x7fd6fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.110+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd70c0a09d0 con 0x7fd70c15e200 2024-03-24T10:34:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.290+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd70c0a09d0 con 0x7fd70c15e200 2024-03-24T10:34:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.291+0000 7fd6f3fff640 1 -- 172.21.15.1:0/3559710059 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fd70c0a09d0 con 0x7fd70c15e200 2024-03-24T10:34:32.291 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 msgr2=0x7fd6f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 0x7fd6f806dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc036000 tx=0x7fd6fc008030 comp rx=0 tx=0).stop 2024-03-24T10:34:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 msgr2=0x7fd70c09f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c700 tx=0x7fd70400cbc0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 shutdown_connections 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd6f806b810 0x7fd6f806dcd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd70c15e200 0x7fd70c09f300 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd70c0d0e30 0x7fd70c0a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 --2- 172.21.15.1:0/3559710059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd70c0cf820 0x7fd70c1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 >> 172.21.15.1:0/3559710059 conn(0x7fd70c0d3d30 msgr2=0x7fd70c0b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 shutdown_connections 2024-03-24T10:34:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.293+0000 7fd710ba0640 1 -- 172.21.15.1:0/3559710059 wait complete. 2024-03-24T10:34:32.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:32.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 Processor -- start 2024-03-24T10:34:32.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 -- start start 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f30440b4e40 0x7f30440d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30440c7a40 con 0x7f3044162610 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30440b55b0 con 0x7f30441615f0 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.409+0000 7f3049177640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f30440cfd50 con 0x7f30440b4e40 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f304359e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3044162610 0x7f30440d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52804/0 (socket says 172.21.15.1:52804) 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f304359e640 1 -- 172.21.15.1:0/1774411100 learned_addr learned my addr 172.21.15.1:0/1774411100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:32.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f303bfff640 1 --2- 172.21.15.1:0/1774411100 >> v2:172.21.15.3:3300/0 conn(0x7f30440b4e40 0x7f30440d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913567143 0 0) 0x7f30440c7a40 con 0x7f3044162610 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3030003520 con 0x7f3044162610 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f303bfff640 1 -- 172.21.15.1:0/1774411100 >> v1:172.21.15.3:6789/0 conn(0x7f30441615f0 legacy=0x7f3044161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30491421 0 0) 0x7f30440b55b0 con 0x7f30441615f0 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f303bfff640 1 -- 172.21.15.1:0/1774411100 >> v1:172.21.15.1:6789/0 conn(0x7f3044162610 legacy=0x7f30440d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.410+0000 7f303bfff640 1 -- 172.21.15.1:0/1774411100 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30440b52d0 con 0x7f30440b4e40 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f303bfff640 1 --2- 172.21.15.1:0/1774411100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30440b4e40 0x7f30440d4a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f302c0097a0 tx=0x7f302c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34b080318cd9526 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c030610 con 0x7f30440b4e40 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f302c030770 con 0x7f30440b4e40 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f304259c640 1 -- 172.21.15.1:0/1774411100 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c030ae0 con 0x7f30440b4e40 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 -- 172.21.15.1:0/1774411100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30440b4e40 msgr2=0x7f30440d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 --2- 172.21.15.1:0/1774411100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30440b4e40 0x7f30440d4a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f302c0097a0 tx=0x7f302c02b960 comp rx=0 tx=0).stop 2024-03-24T10:34:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 -- 172.21.15.1:0/1774411100 shutdown_connections 2024-03-24T10:34:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 --2- 172.21.15.1:0/1774411100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30440b4e40 0x7f30440d4a10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 -- 172.21.15.1:0/1774411100 >> 172.21.15.1:0/1774411100 conn(0x7f30440d3ef0 msgr2=0x7f30440d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.411+0000 7f3049177640 1 -- 172.21.15.1:0/1774411100 shutdown_connections 2024-03-24T10:34:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.412+0000 7f3049177640 1 -- 172.21.15.1:0/1774411100 wait complete. 2024-03-24T10:34:32.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.413+0000 7f3049177640 1 Processor -- start 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.413+0000 7f3049177640 1 -- start start 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 0x7f30440cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3044156080 0x7f304415b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f30440cbcc0 con 0x7f30441615f0 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f30440cc0e0 con 0x7f3044156080 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f30440ce2e0 con 0x7f3044162610 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43376/0 (socket says 172.21.15.1:43376) 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 -- 172.21.15.1:0/785743011 learned_addr learned my addr 172.21.15.1:0/785743011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f303bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 0x7f30440cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f304359e640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3044156080 0x7f304415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 -- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 msgr2=0x7f30440cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 0x7f30440cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 -- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3044156080 msgr2=0x7f304415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3044156080 0x7f304415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 -- 172.21.15.1:0/785743011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c009400 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f303bfff640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 0x7f30440cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3042d9d640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f303400c4e0 tx=0x7f303400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034005480 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30340035f0 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3044159c90 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340039c0 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.414+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30440cb8e0 con 0x7f30441615f0 2024-03-24T10:34:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.415+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30440d1910 con 0x7f30441615f0 2024-03-24T10:34:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.416+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3034003b20 con 0x7f30441615f0 2024-03-24T10:34:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.416+0000 7f303b7fe640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 0x7f302406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.416+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3034089660 con 0x7f30441615f0 2024-03-24T10:34:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.416+0000 7f303bfff640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 0x7f302406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.417+0000 7f303bfff640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 0x7f302406dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f302c036000 tx=0x7f302c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.421+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3034054120 con 0x7f30441615f0 2024-03-24T10:34:32.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.600+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f304404f470 con 0x7f30441615f0 2024-03-24T10:34:32.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.601+0000 7f303b7fe640 1 -- 172.21.15.1:0/785743011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f30340577e0 con 0x7f30441615f0 2024-03-24T10:34:32.601 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 msgr2=0x7f302406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 0x7f302406dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f302c036000 tx=0x7f302c008030 comp rx=0 tx=0).stop 2024-03-24T10:34:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 msgr2=0x7f30440b97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f303400c4e0 tx=0x7f303400c9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 shutdown_connections 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f302406b810 0x7f302406dcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3044156080 0x7f304415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3044162610 0x7f30440cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 --2- 172.21.15.1:0/785743011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30441615f0 0x7f30440b97c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 >> 172.21.15.1:0/785743011 conn(0x7f30440d3ef0 msgr2=0x7f30440d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 shutdown_connections 2024-03-24T10:34:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.603+0000 7f3049177640 1 -- 172.21.15.1:0/785743011 wait complete. 2024-03-24T10:34:32.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:32.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.718+0000 7fc2e4415640 1 Processor -- start 2024-03-24T10:34:32.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.718+0000 7fc2e4415640 1 -- start start 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e4415640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e4415640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2dc15c4d0 0x7fc2dc15c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e4415640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc06eed0 con 0x7fc2dc0b4570 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e4415640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0b34d0 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e4415640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b37b0 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2dc15c4d0 0x7fc2dc15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e21b1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2dc15c4d0 0x7fc2dc15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:48820/0 (socket says 172.21.15.1:48820) 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e19b0640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc2dc0b4570 0x7fc2dc0b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:34032/0 (socket says 172.21.15.1:34032) 2024-03-24T10:34:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e21b1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:45566/0 (socket says 172.21.15.1:45566) 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 -- 172.21.15.1:0/3069589937 learned_addr learned my addr 172.21.15.1:0/3069589937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 -- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 msgr2=0x7fc2dc06e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e29b2640 1 -- 172.21.15.1:0/3069589937 >> v1:172.21.15.3:6789/0 conn(0x7fc2dc0b4570 legacy=0x7fc2dc0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.719+0000 7fc2e11af640 1 -- 172.21.15.1:0/3069589937 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321937902 0 0) 0x7fc2dc06eed0 con 0x7fc2dc0b4570 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.720+0000 7fc2e29b2640 1 -- 172.21.15.1:0/3069589937 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc0b3a90 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.720+0000 7fc2e21b1640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.720+0000 7fc2e29b2640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc15c960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009e80 tx=0x7fc2cc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec5a61759203110a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.720+0000 7fc2e11af640 1 -- 172.21.15.1:0/3069589937 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc02caf0 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.720+0000 7fc2e11af640 1 -- 172.21.15.1:0/3069589937 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2cc02cc50 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e11af640 1 -- 172.21.15.1:0/3069589937 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0343f0 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 -- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 msgr2=0x7fc2dc15c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc15c960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009e80 tx=0x7fc2cc02bf50 comp rx=0 tx=0).stop 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 -- 172.21.15.1:0/3069589937 shutdown_connections 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc15c960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 --2- 172.21.15.1:0/3069589937 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc06e5b0 0x7fc2dc06e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 -- 172.21.15.1:0/3069589937 >> 172.21.15.1:0/3069589937 conn(0x7fc2dc0b1710 msgr2=0x7fc2dc0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 -- 172.21.15.1:0/3069589937 shutdown_connections 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.721+0000 7fc2e4415640 1 -- 172.21.15.1:0/3069589937 wait complete. 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.722+0000 7fc2e4415640 1 Processor -- start 2024-03-24T10:34:32.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.722+0000 7fc2e4415640 1 -- start start 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 0x7fc2dc0cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc13a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc15a480 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc13b7b0 con 0x7fc2dc15c4d0 2024-03-24T10:34:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e4415640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc13ba90 con 0x7fc2dc0b4570 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43386/0 (socket says 172.21.15.1:43386) 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 -- 172.21.15.1:0/566574039 learned_addr learned my addr 172.21.15.1:0/566574039 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e29b2640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e19b0640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 0x7fc2dc0cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 -- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 msgr2=0x7fc2dc0cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 0x7fc2dc0cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 -- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 msgr2=0x7fc2dc13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc13a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e21b1640 1 -- 172.21.15.1:0/566574039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d4009400 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e29b2640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.723+0000 7fc2e19b0640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 0x7fc2dc0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2e21b1640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400e960 tx=0x7fc2d400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400ca00 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d400cb60 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2cc009b30 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4013470 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.724+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc13c050 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.725+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc04f4e0 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.725+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc2d40135d0 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.725+0000 7fc2daffd640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 0x7fc2c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.726+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc2d408a360 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.726+0000 7fc2e19b0640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 0x7fc2c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.726+0000 7fc2e19b0640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 0x7fc2c806dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc13f980 tx=0x7fc2d0007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.730+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc2d4054e30 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.910+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2dc0cd5a0 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.910+0000 7fc2daffd640 1 -- 172.21.15.1:0/566574039 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fc2d40584f0 con 0x7fc2dc06e5b0 2024-03-24T10:34:32.910 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 msgr2=0x7fc2c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 0x7fc2c806dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc13f980 tx=0x7fc2d0007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 msgr2=0x7fc2dc159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400e960 tx=0x7fc2d400ee20 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 shutdown_connections 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc2c806b840 0x7fc2c806dd00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2dc15c4d0 0x7fc2dc13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2dc0b4570 0x7fc2dc0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 --2- 172.21.15.1:0/566574039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2dc06e5b0 0x7fc2dc159f40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 >> 172.21.15.1:0/566574039 conn(0x7fc2dc0b1710 msgr2=0x7fc2dc15b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 shutdown_connections 2024-03-24T10:34:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:32.912+0000 7fc2e4415640 1 -- 172.21.15.1:0/566574039 wait complete. 2024-03-24T10:34:32.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.025+0000 7fa90dfa5640 1 Processor -- start 2024-03-24T10:34:33.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.025+0000 7fa90dfa5640 1 -- start start 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa90dfa5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9080ce540 0x7fa9080ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa90dfa5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa90dfa5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9080b13a0 con 0x7fa90815e540 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa90dfa5640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa9080d1450 con 0x7fa9080ce540 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa90dfa5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa90806c4e0 con 0x7fa9080d27e0 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9080ce540 0x7fa9080ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9080ce540 0x7fa9080ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43402/0 (socket says 172.21.15.1:43402) 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 -- 172.21.15.1:0/1056287953 learned_addr learned my addr 172.21.15.1:0/1056287953 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa907fff640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 -- 172.21.15.1:0/1056287953 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 msgr2=0x7fa9080d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 -- 172.21.15.1:0/1056287953 >> v1:172.21.15.3:6789/0 conn(0x7fa90815e540 legacy=0x7fa9080d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9067fc640 1 -- 172.21.15.1:0/1056287953 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747074481 0 0) 0x7fa9080b13a0 con 0x7fa90815e540 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.026+0000 7fa9077fe640 1 -- 172.21.15.1:0/1056287953 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90806ca10 con 0x7fa9080ce540 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.027+0000 7fa9077fe640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9080ce540 0x7fa9080ce920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e0f92a1e8c506f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.027+0000 7fa9067fc640 1 -- 172.21.15.1:0/1056287953 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8030840 con 0x7fa9080ce540 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.027+0000 7fa9067fc640 1 -- 172.21.15.1:0/1056287953 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f80309a0 con 0x7fa9080ce540 2024-03-24T10:34:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.027+0000 7fa9067fc640 1 -- 172.21.15.1:0/1056287953 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8030d30 con 0x7fa9080ce540 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 -- 172.21.15.1:0/1056287953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9080ce540 msgr2=0x7fa9080ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9080ce540 0x7fa9080ce920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f802b980 comp rx=0 tx=0).stop 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 -- 172.21.15.1:0/1056287953 shutdown_connections 2024-03-24T10:34:33.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/1056287953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9080ce540 0x7fa9080ce920 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 -- 172.21.15.1:0/1056287953 >> 172.21.15.1:0/1056287953 conn(0x7fa9080d1cc0 msgr2=0x7fa9080d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 -- 172.21.15.1:0/1056287953 shutdown_connections 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.028+0000 7fa90dfa5640 1 -- 172.21.15.1:0/1056287953 wait complete. 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.029+0000 7fa90dfa5640 1 Processor -- start 2024-03-24T10:34:33.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.029+0000 7fa90dfa5640 1 -- start start 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9080ce540 0x7fa9080c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa9080c9a70 con 0x7fa90815e540 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa9080ca810 con 0x7fa9080ce540 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa90dfa5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa9080c9c90 con 0x7fa9080d27e0 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43416/0 (socket says 172.21.15.1:43416) 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 -- 172.21.15.1:0/3223896720 learned_addr learned my addr 172.21.15.1:0/3223896720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa906ffd640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa9077fe640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9080ce540 0x7fa9080c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 -- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 msgr2=0x7fa9080c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 -- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9080ce540 msgr2=0x7fa9080c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9080ce540 0x7fa9080c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa907fff640 1 -- 172.21.15.1:0/3223896720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f80094d0 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.030+0000 7fa906ffd640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa907fff640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa90000efc0 tx=0x7fa90000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900014480 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90000f040 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9080c9f70 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900014820 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.031+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9080ca530 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.032+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90814b0a0 con 0x7fa90815e540 2024-03-24T10:34:33.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.032+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa900014980 con 0x7fa90815e540 2024-03-24T10:34:33.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.032+0000 7fa8effff640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 0x7fa8f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.033+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa90008ae60 con 0x7fa90815e540 2024-03-24T10:34:33.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.033+0000 7fa9077fe640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 0x7fa8f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.033+0000 7fa9077fe640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 0x7fa8f406dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8036000 tx=0x7fa8f8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.037+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9000559c0 con 0x7fa90815e540 2024-03-24T10:34:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.230+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa90804f4e0 con 0x7fa90815e540 2024-03-24T10:34:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.230+0000 7fa8effff640 1 -- 172.21.15.1:0/3223896720 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fa900059080 con 0x7fa90815e540 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 msgr2=0x7fa8f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 0x7fa8f406dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8036000 tx=0x7fa8f8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 msgr2=0x7fa9081f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa90000efc0 tx=0x7fa90000c490 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 shutdown_connections 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa8f406b810 0x7fa8f406dcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa90815e540 0x7fa9081f2be0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa9080d27e0 0x7fa9080c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 --2- 172.21.15.1:0/3223896720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9080ce540 0x7fa9080c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 >> 172.21.15.1:0/3223896720 conn(0x7fa9080d1cc0 msgr2=0x7fa90815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 shutdown_connections 2024-03-24T10:34:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.232+0000 7fa90dfa5640 1 -- 172.21.15.1:0/3223896720 wait complete. 2024-03-24T10:34:33.243 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi003.front.sepia.ceph.com... 2024-03-24T10:34:33.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:33.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-24T10:34:33.275 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:33.275 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:34:33.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:33.275 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T10:34:33.341 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:33.341 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:33.357 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:33.357 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:33.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.553+0000 7fe9a4c47640 1 Processor -- start 2024-03-24T10:34:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.553+0000 7fe9a4c47640 1 -- start start 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe9a4c47640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe9a4c47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe9a4c47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a015e770 con 0x7fe9a00d1000 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe9a4c47640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a00b3190 con 0x7fe9a015e390 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe9a4c47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a00d4e40 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:45598/0 (socket says 172.21.15.1:45598) 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:48844/0 (socket says 172.21.15.1:48844) 2024-03-24T10:34:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 -- 172.21.15.1:0/2575110318 learned_addr learned my addr 172.21.15.1:0/2575110318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99dd9b640 1 --1- 172.21.15.1:0/2575110318 >> v1:172.21.15.3:6789/0 conn(0x7fe9a00d1000 0x7fe9a00cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:34052/0 (socket says 172.21.15.1:34052) 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 -- 172.21.15.1:0/2575110318 >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 msgr2=0x7fe9a00d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 --2- 172.21.15.1:0/2575110318 >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 -- 172.21.15.1:0/2575110318 >> v1:172.21.15.3:6789/0 conn(0x7fe9a00d1000 legacy=0x7fe9a00cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99ed9d640 1 -- 172.21.15.1:0/2575110318 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a00d1e40 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.554+0000 7fe99e59c640 1 --2- 172.21.15.1:0/2575110318 >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.555+0000 7fe99ed9d640 1 --2- 172.21.15.1:0/2575110318 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe990009e50 tx=0x7fe99002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0c560ce19b4d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.555+0000 7fe99d59a640 1 -- 172.21.15.1:0/2575110318 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99002c9e0 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.555+0000 7fe99d59a640 1 -- 172.21.15.1:0/2575110318 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99002cb40 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.555+0000 7fe99d59a640 1 -- 172.21.15.1:0/2575110318 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9900343f0 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 -- 172.21.15.1:0/2575110318 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 msgr2=0x7fe9a00d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/2575110318 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe990009e50 tx=0x7fe99002be40 comp rx=0 tx=0).stop 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 -- 172.21.15.1:0/2575110318 shutdown_connections 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/2575110318 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a00d29e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/2575110318 >> v2:172.21.15.3:3300/0 conn(0x7fe9a015e390 0x7fe9a00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 -- 172.21.15.1:0/2575110318 >> 172.21.15.1:0/2575110318 conn(0x7fe9a00d3ec0 msgr2=0x7fe9a00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 -- 172.21.15.1:0/2575110318 shutdown_connections 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.556+0000 7fe9a4c47640 1 -- 172.21.15.1:0/2575110318 wait complete. 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.557+0000 7fe9a4c47640 1 Processor -- start 2024-03-24T10:34:33.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.557+0000 7fe9a4c47640 1 -- start start 2024-03-24T10:34:33.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 0x7fe9a009f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a00cb780 con 0x7fe9a00d1000 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a009f000 con 0x7fe9a015e390 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe9a4c47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a00cb130 con 0x7fe9a00b4ba0 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99e59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 0x7fe9a009f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99e59c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:45608/0 (socket says 172.21.15.1:45608) 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99e59c640 1 -- 172.21.15.1:0/624834027 learned_addr learned my addr 172.21.15.1:0/624834027 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.558+0000 7fe99dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43430/0 (socket says 172.21.15.1:43430) 2024-03-24T10:34:33.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 -- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 msgr2=0x7fe9a0155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 -- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 msgr2=0x7fe9a009f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 0x7fe9a009f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 -- 172.21.15.1:0/624834027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe998009400 con 0x7fe9a00d1000 2024-03-24T10:34:33.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99e59c640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99ed9d640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 0x7fe9a009f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe99dd9b640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe988000f60 tx=0x7fe98800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880074e0 con 0x7fe9a00d1000 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe988007660 con 0x7fe9a00d1000 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe990009b30 con 0x7fe9a00d1000 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988004a20 con 0x7fe9a00d1000 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.559+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a01f3960 con 0x7fe9a00d1000 2024-03-24T10:34:33.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.560+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a00caf70 con 0x7fe9a00d1000 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.560+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe988004b80 con 0x7fe9a00d1000 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.561+0000 7fe996ffd640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 0x7fe98006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.561+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe988089720 con 0x7fe9a00d1000 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.561+0000 7fe99e59c640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 0x7fe98006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.561+0000 7fe99e59c640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 0x7fe98006dd00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe9980114b0 tx=0x7fe998009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.565+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe988054280 con 0x7fe9a00d1000 2024-03-24T10:34:33.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.749+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9a015e770 con 0x7fe9a00d1000 2024-03-24T10:34:33.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.750+0000 7fe996ffd640 1 -- 172.21.15.1:0/624834027 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16473 (secure 0 0 0) 0x7fe9a015e770 con 0x7fe9a00d1000 2024-03-24T10:34:33.750 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:33.750 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:33:56.119950+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 msgr2=0x7fe98006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 0x7fe98006dd00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe9980114b0 tx=0x7fe998009190 comp rx=0 tx=0).stop 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 msgr2=0x7fe9a009e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe988000f60 tx=0x7fe98800c800 comp rx=0 tx=0).stop 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 shutdown_connections 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe98006b840 0x7fe98006dd00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9a015e390 0x7fe9a009f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9a00d1000 0x7fe9a009e620 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 --2- 172.21.15.1:0/624834027 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe9a00b4ba0 0x7fe9a0155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 >> 172.21.15.1:0/624834027 conn(0x7fe9a00d3ec0 msgr2=0x7fe9a00b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 shutdown_connections 2024-03-24T10:34:33.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.752+0000 7fe9a4c47640 1 -- 172.21.15.1:0/624834027 wait complete. 2024-03-24T10:34:33.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:33.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.868+0000 7fd30c62a640 1 Processor -- start 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.868+0000 7fd30c62a640 1 -- start start 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 0x7fd30406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3040b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd3040b40c0 0x7fd30415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd304070bd0 con 0x7fd304154be0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd304154fc0 con 0x7fd3040b40c0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30c62a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd304071370 con 0x7fd3040cf450 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30a3c6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 0x7fd30406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3040b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30abc7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd3040b40c0 0x7fd30415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30a3c6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 0x7fd30406d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43436/0 (socket says 172.21.15.1:43436) 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd30a3c6640 1 -- 172.21.15.1:0/3545856474 learned_addr learned my addr 172.21.15.1:0/3545856474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 -- 172.21.15.1:0/3545856474 >> v2:172.21.15.3:3300/0 conn(0x7fd3040b40c0 msgr2=0x7fd30415aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 --2- 172.21.15.1:0/3545856474 >> v2:172.21.15.3:3300/0 conn(0x7fd3040b40c0 0x7fd30415aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 -- 172.21.15.1:0/3545856474 >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 msgr2=0x7fd30406d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 --2- 172.21.15.1:0/3545856474 >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 0x7fd30406d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.869+0000 7fd309bc5640 1 -- 172.21.15.1:0/3545856474 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040b36a0 con 0x7fd3040cf450 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd309bc5640 1 --2- 172.21.15.1:0/3545856474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3040b3b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ba40 tx=0x7fd2f402df80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f710ac6bf310eac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd3093c4640 1 -- 172.21.15.1:0/3545856474 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f402ed30 con 0x7fd3040cf450 2024-03-24T10:34:33.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd3093c4640 1 -- 172.21.15.1:0/3545856474 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f402ee90 con 0x7fd3040cf450 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd3093c4640 1 -- 172.21.15.1:0/3545856474 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4036660 con 0x7fd3040cf450 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd30c62a640 1 -- 172.21.15.1:0/3545856474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 msgr2=0x7fd3040b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.870+0000 7fd30c62a640 1 --2- 172.21.15.1:0/3545856474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3040b3b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ba40 tx=0x7fd2f402df80 comp rx=0 tx=0).stop 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 -- 172.21.15.1:0/3545856474 shutdown_connections 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 --2- 172.21.15.1:0/3545856474 >> v2:172.21.15.3:3300/0 conn(0x7fd3040b40c0 0x7fd30415aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 --2- 172.21.15.1:0/3545856474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3040b3b80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 --2- 172.21.15.1:0/3545856474 >> v2:172.21.15.1:3300/0 conn(0x7fd304154be0 0x7fd30406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 -- 172.21.15.1:0/3545856474 >> 172.21.15.1:0/3545856474 conn(0x7fd3040b1710 msgr2=0x7fd3040b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:33.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 -- 172.21.15.1:0/3545856474 shutdown_connections 2024-03-24T10:34:33.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.871+0000 7fd30c62a640 1 -- 172.21.15.1:0/3545856474 wait complete. 2024-03-24T10:34:33.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 Processor -- start 2024-03-24T10:34:33.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 -- start start 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3041e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd304154be0 0x7fd30414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041efac0 con 0x7fd3040b40c0 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041521c0 con 0x7fd304154be0 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.872+0000 7fd30c62a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd304152460 con 0x7fd3040cf450 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43450/0 (socket says 172.21.15.1:43450) 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 -- 172.21.15.1:0/488119798 learned_addr learned my addr 172.21.15.1:0/488119798 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd309bc5640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3041e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30abc7640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd304154be0 0x7fd30414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 -- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 msgr2=0x7fd3041e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3041e42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 -- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd304154be0 msgr2=0x7fd30414f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd304154be0 0x7fd30414f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 -- 172.21.15.1:0/488119798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f400b6a0 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd309bc5640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3041e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30a3c6640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd300009fd0 tx=0x7fd30000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30000c940 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd30000caa0 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd304152740 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30000ce30 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.873+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd304152d00 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.874+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30404f4e0 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.875+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd300013470 con 0x7fd3040b40c0 2024-03-24T10:34:33.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.875+0000 7fd2faffd640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 0x7fd2e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:33.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.875+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd30008a1e0 con 0x7fd3040b40c0 2024-03-24T10:34:33.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.875+0000 7fd309bc5640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 0x7fd2e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:33.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.876+0000 7fd309bc5640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 0x7fd2e806dd00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4038000 tx=0x7fd2f4032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:33.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:33.879+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd300054d40 con 0x7fd3040b40c0 2024-03-24T10:34:34.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.061+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd30415d490 con 0x7fd3040b40c0 2024-03-24T10:34:34.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.062+0000 7fd2faffd640 1 -- 172.21.15.1:0/488119798 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fd300058400 con 0x7fd3040b40c0 2024-03-24T10:34:34.062 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 msgr2=0x7fd2e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 0x7fd2e806dd00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4038000 tx=0x7fd2f4032000 comp rx=0 tx=0).stop 2024-03-24T10:34:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 msgr2=0x7fd304151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd300009fd0 tx=0x7fd30000ed40 comp rx=0 tx=0).stop 2024-03-24T10:34:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 shutdown_connections 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd2e806b840 0x7fd2e806dd00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd304154be0 0x7fd30414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3040cf450 0x7fd3041e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 --2- 172.21.15.1:0/488119798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3040b40c0 0x7fd304151600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 >> 172.21.15.1:0/488119798 conn(0x7fd3040b1710 msgr2=0x7fd3040b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 shutdown_connections 2024-03-24T10:34:34.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.064+0000 7fd30c62a640 1 -- 172.21.15.1:0/488119798 wait complete. 2024-03-24T10:34:34.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.182+0000 7faa6182a640 1 Processor -- start 2024-03-24T10:34:34.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.182+0000 7faa6182a640 1 -- start start 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa6182a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa5c0d0690 0x7faa5c165bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa6182a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c0b45c0 con 0x7faa5c0cbf90 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa6182a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5c15ed00 con 0x7faa5c15e860 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa6182a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7faa5c06c930 con 0x7faa5c0d0690 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa5c0cbf90 0x7faa5c15bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52818/0 (socket says 172.21.15.1:52818) 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5affd640 1 -- 172.21.15.1:0/3987000860 learned_addr learned my addr 172.21.15.1:0/3987000860 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5b7fe640 1 --2- 172.21.15.1:0/3987000860 >> v2:172.21.15.3:3300/0 conn(0x7faa5c0d0690 0x7faa5c165bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585215058 0 0) 0x7faa5c0b45c0 con 0x7faa5c0cbf90 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa48003520 con 0x7faa5c0cbf90 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5b7fe640 1 -- 172.21.15.1:0/3987000860 >> v1:172.21.15.3:6789/0 conn(0x7faa5c15e860 legacy=0x7faa5c0cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373678970 0 0) 0x7faa5c15ed00 con 0x7faa5c15e860 2024-03-24T10:34:34.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5b7fe640 1 -- 172.21.15.1:0/3987000860 >> v1:172.21.15.1:6789/0 conn(0x7faa5c0cbf90 legacy=0x7faa5c15bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.183+0000 7faa5b7fe640 1 -- 172.21.15.1:0/3987000860 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c15efe0 con 0x7faa5c0d0690 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.184+0000 7faa5b7fe640 1 --2- 172.21.15.1:0/3987000860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c0d0690 0x7faa5c165bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faa540097a0 tx=0x7faa5402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7eca92064f7e275 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.184+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54031830 con 0x7faa5c0d0690 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.184+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa54031990 con 0x7faa5c0d0690 2024-03-24T10:34:34.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.184+0000 7faa59ffb640 1 -- 172.21.15.1:0/3987000860 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54031d20 con 0x7faa5c0d0690 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 -- 172.21.15.1:0/3987000860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c0d0690 msgr2=0x7faa5c165bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 --2- 172.21.15.1:0/3987000860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c0d0690 0x7faa5c165bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faa540097a0 tx=0x7faa5402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 -- 172.21.15.1:0/3987000860 shutdown_connections 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 --2- 172.21.15.1:0/3987000860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c0d0690 0x7faa5c165bf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 -- 172.21.15.1:0/3987000860 >> 172.21.15.1:0/3987000860 conn(0x7faa5c06b4f0 msgr2=0x7faa5c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 -- 172.21.15.1:0/3987000860 shutdown_connections 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.185+0000 7faa6182a640 1 -- 172.21.15.1:0/3987000860 wait complete. 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.186+0000 7faa6182a640 1 Processor -- start 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.186+0000 7faa6182a640 1 -- start start 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa5c0d0690 0x7faa5c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c15e860 0x7faa5c155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c1f1220 con 0x7faa5c0cbf90 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c1f1500 con 0x7faa5c15e860 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa6182a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faa5c159890 con 0x7faa5c0d0690 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43454/0 (socket says 172.21.15.1:43454) 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 -- 172.21.15.1:0/2840024388 learned_addr learned my addr 172.21.15.1:0/2840024388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5b7fe640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c15e860 0x7faa5c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5a7fc640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa5c0d0690 0x7faa5c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 -- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa5c0d0690 msgr2=0x7faa5c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa5c0d0690 0x7faa5c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 -- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c15e860 msgr2=0x7faa5c155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c15e860 0x7faa5c155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.187+0000 7faa5affd640 1 -- 172.21.15.1:0/2840024388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54009400 con 0x7faa5c0cbf90 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa5affd640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00cb40 tx=0x7faa4c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c00aca0 con 0x7faa5c0cbf90 2024-03-24T10:34:34.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4c00ae00 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5c159b70 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c012620 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.188+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5c1594b0 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.189+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5c04f470 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.189+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa4c007070 con 0x7faa5c0cbf90 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.190+0000 7faa53fff640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 0x7faa3c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.190+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faa4c08a110 con 0x7faa5c0cbf90 2024-03-24T10:34:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.190+0000 7faa5a7fc640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 0x7faa3c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.190+0000 7faa5a7fc640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 0x7faa3c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faa5c154e20 tx=0x7faa44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.194+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa4c054c70 con 0x7faa5c0cbf90 2024-03-24T10:34:34.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.374+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faa5c1536f0 con 0x7faa5c0cbf90 2024-03-24T10:34:34.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.374+0000 7faa53fff640 1 -- 172.21.15.1:0/2840024388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7faa4c017020 con 0x7faa5c0cbf90 2024-03-24T10:34:34.375 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 msgr2=0x7faa3c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 0x7faa3c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faa5c154e20 tx=0x7faa44009040 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 msgr2=0x7faa5c150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00cb40 tx=0x7faa4c00a060 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 shutdown_connections 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faa3c06b840 0x7faa3c06dd00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa5c15e860 0x7faa5c155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa5c0d0690 0x7faa5c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 --2- 172.21.15.1:0/2840024388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa5c0cbf90 0x7faa5c150430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 >> 172.21.15.1:0/2840024388 conn(0x7faa5c06b4f0 msgr2=0x7faa5c153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 shutdown_connections 2024-03-24T10:34:34.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.376+0000 7faa6182a640 1 -- 172.21.15.1:0/2840024388 wait complete. 2024-03-24T10:34:34.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:34.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 Processor -- start 2024-03-24T10:34:34.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 -- start start 2024-03-24T10:34:34.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03f8158540 0x7f03f806b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f03f806b5e0 0x7f03f80af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f8072760 con 0x7f03f80aff30 2024-03-24T10:34:34.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f03f806ba60 con 0x7f03f8158540 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.493+0000 7f03fcb25640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f03f80b03c0 con 0x7f03f806b5e0 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03f8158540 0x7f03f806b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03f80aff30 0x7f03f80ae950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52822/0 (socket says 172.21.15.1:52822) 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f6d9d640 1 -- 172.21.15.1:0/127007624 learned_addr learned my addr 172.21.15.1:0/127007624 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03effff640 1 --2- 172.21.15.1:0/127007624 >> v2:172.21.15.3:3300/0 conn(0x7f03f806b5e0 0x7f03f80af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f659c640 1 -- 172.21.15.1:0/127007624 >> v2:172.21.15.3:3300/0 conn(0x7f03f806b5e0 msgr2=0x7f03f80af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f659c640 1 --2- 172.21.15.1:0/127007624 >> v2:172.21.15.3:3300/0 conn(0x7f03f806b5e0 0x7f03f80af9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f659c640 1 -- 172.21.15.1:0/127007624 >> v1:172.21.15.1:6789/0 conn(0x7f03f80aff30 legacy=0x7f03f80ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f659c640 1 -- 172.21.15.1:0/127007624 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f80b05d0 con 0x7f03f8158540 2024-03-24T10:34:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.494+0000 7f03f5d9b640 1 -- 172.21.15.1:0/127007624 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207754428 0 0) 0x7f03f8072760 con 0x7f03f80aff30 2024-03-24T10:34:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03f659c640 1 --2- 172.21.15.1:0/127007624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f806b0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f03f806f4a0 tx=0x7f03e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed8300e8b16f939 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03f5d9b640 1 -- 172.21.15.1:0/127007624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e802e070 con 0x7f03f8158540 2024-03-24T10:34:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03f5d9b640 1 -- 172.21.15.1:0/127007624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e80315a0 con 0x7f03f8158540 2024-03-24T10:34:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03f5d9b640 1 -- 172.21.15.1:0/127007624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8031930 con 0x7f03f8158540 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 -- 172.21.15.1:0/127007624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 msgr2=0x7f03f806b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 --2- 172.21.15.1:0/127007624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f806b0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f03f806f4a0 tx=0x7f03e802b840 comp rx=0 tx=0).stop 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 -- 172.21.15.1:0/127007624 shutdown_connections 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 --2- 172.21.15.1:0/127007624 >> v2:172.21.15.3:3300/0 conn(0x7f03f806b5e0 0x7f03f80af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 --2- 172.21.15.1:0/127007624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f806b0a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 -- 172.21.15.1:0/127007624 >> 172.21.15.1:0/127007624 conn(0x7f03f80b2820 msgr2=0x7f03f80b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 -- 172.21.15.1:0/127007624 shutdown_connections 2024-03-24T10:34:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.495+0000 7f03fcb25640 1 -- 172.21.15.1:0/127007624 wait complete. 2024-03-24T10:34:34.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 Processor -- start 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 -- start start 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 0x7f03f815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 0x7f03f8152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f03f8159740 con 0x7f03f8158540 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f03f815e010 con 0x7f03f806b5e0 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.497+0000 7f03fcb25640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f03f8158ac0 con 0x7f03f80aff30 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43478/0 (socket says 172.21.15.1:43478) 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 -- 172.21.15.1:0/161643514 learned_addr learned my addr 172.21.15.1:0/161643514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f659c640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 0x7f03f815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03effff640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 0x7f03f8152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 -- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 msgr2=0x7f03f8152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 0x7f03f8152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 -- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 msgr2=0x7f03f815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 0x7f03f815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 -- 172.21.15.1:0/161643514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e8009400 con 0x7f03f8158540 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03effff640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 0x7f03f8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f659c640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 0x7f03f815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03f6d9d640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c7a0 tx=0x7f03f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f000a640 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f000a7a0 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f000abd0 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f8158da0 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.498+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f8159360 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.499+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f8159880 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.500+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03f000ad30 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.500+0000 7f03eeffd640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.500+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f03f0089980 con 0x7f03f8158540 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.500+0000 7f03f659c640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.501+0000 7f03f659c640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f03e8036000 tx=0x7f03e803e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.504+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03f8159880 con 0x7f03f8158540 2024-03-24T10:34:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.688+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f03f8159880 con 0x7f03f8158540 2024-03-24T10:34:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.689+0000 7f03eeffd640 1 -- 172.21.15.1:0/161643514 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f03f8159880 con 0x7f03f8158540 2024-03-24T10:34:34.690 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 msgr2=0x7f03d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f03e8036000 tx=0x7f03e803e040 comp rx=0 tx=0).stop 2024-03-24T10:34:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 msgr2=0x7f03f815a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c7a0 tx=0x7f03f000cc60 comp rx=0 tx=0).stop 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 shutdown_connections 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f8158540 0x7f03f815a120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f80aff30 0x7f03f8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 --2- 172.21.15.1:0/161643514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f806b5e0 0x7f03f815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 >> 172.21.15.1:0/161643514 conn(0x7f03f80b2820 msgr2=0x7f03f80d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 shutdown_connections 2024-03-24T10:34:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.691+0000 7f03fcb25640 1 -- 172.21.15.1:0/161643514 wait complete. 2024-03-24T10:34:34.702 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.805+0000 7f25342bb640 1 Processor -- start 2024-03-24T10:34:34.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.805+0000 7f25342bb640 1 -- start start 2024-03-24T10:34:34.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f252c15e390 0x7f252c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f252c0d1000 0x7f252c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 0x7f252c0cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f252c0b3190 con 0x7f252c0d1000 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f252c15e770 con 0x7f252c15e390 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f25342bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f252c0d4e40 con 0x7f252c0b3320 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f2531856640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f252c0d1000 0x7f252c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f2531856640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f252c0d1000 0x7f252c0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43486/0 (socket says 172.21.15.1:43486) 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f2531856640 1 -- 172.21.15.1:0/2168169841 learned_addr learned my addr 172.21.15.1:0/2168169841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.806+0000 7f2532858640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 0x7f252c0cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2532057640 1 --2- 172.21.15.1:0/2168169841 >> v2:172.21.15.3:3300/0 conn(0x7f252c15e390 0x7f252c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2531856640 1 -- 172.21.15.1:0/2168169841 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 msgr2=0x7f252c0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2531856640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 0x7f252c0cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2531856640 1 -- 172.21.15.1:0/2168169841 >> v2:172.21.15.3:3300/0 conn(0x7f252c15e390 msgr2=0x7f252c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2531856640 1 --2- 172.21.15.1:0/2168169841 >> v2:172.21.15.3:3300/0 conn(0x7f252c15e390 0x7f252c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2531856640 1 -- 172.21.15.1:0/2168169841 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c0d1e40 con 0x7f252c0d1000 2024-03-24T10:34:34.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.807+0000 7f2532858640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 0x7f252c0cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f2531856640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0d29e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f252c0d3040 tx=0x7f251c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4decf2d6c6c30a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f2531055640 1 -- 172.21.15.1:0/2168169841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c031750 con 0x7f252c0d1000 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f2531055640 1 -- 172.21.15.1:0/2168169841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c0318b0 con 0x7f252c0d1000 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f2531055640 1 -- 172.21.15.1:0/2168169841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c031c40 con 0x7f252c0d1000 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 -- 172.21.15.1:0/2168169841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 msgr2=0x7f252c0d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0d29e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f252c0d3040 tx=0x7f251c02b7f0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 -- 172.21.15.1:0/2168169841 shutdown_connections 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c0b3320 0x7f252c0cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 --2- 172.21.15.1:0/2168169841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0d29e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 --2- 172.21.15.1:0/2168169841 >> v2:172.21.15.3:3300/0 conn(0x7f252c15e390 0x7f252c0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 -- 172.21.15.1:0/2168169841 >> 172.21.15.1:0/2168169841 conn(0x7f252c0d3ec0 msgr2=0x7f252c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 -- 172.21.15.1:0/2168169841 shutdown_connections 2024-03-24T10:34:34.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.808+0000 7f25342bb640 1 -- 172.21.15.1:0/2168169841 wait complete. 2024-03-24T10:34:34.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.809+0000 7f25342bb640 1 Processor -- start 2024-03-24T10:34:34.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 -- start start 2024-03-24T10:34:34.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f252c0b3320 0x7f252c155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c15e390 0x7f252c09f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f252c0b77c0 con 0x7f252c0d1000 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f252c1558b0 con 0x7f252c0b3320 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f25342bb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f252c0b6b40 con 0x7f252c15e390 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f2531856640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f2531856640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43494/0 (socket says 172.21.15.1:43494) 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f2531856640 1 -- 172.21.15.1:0/4183897588 learned_addr learned my addr 172.21.15.1:0/4183897588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f2532057640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f252c0b3320 0x7f252c155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.810+0000 7f2532858640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c15e390 0x7f252c09f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 -- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c15e390 msgr2=0x7f252c09f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c15e390 0x7f252c09f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 -- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f252c0b3320 msgr2=0x7f252c155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f252c0b3320 0x7f252c155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 -- 172.21.15.1:0/4183897588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2528009a70 con 0x7f252c0d1000 2024-03-24T10:34:34.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2531856640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f251c02b830 tx=0x7f251c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0338d0 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c033a30 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251c0094d0 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c033de0 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.811+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252c0b7100 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.812+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252c0b7900 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.812+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f251c03a040 con 0x7f252c0d1000 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.813+0000 7f2522ffd640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 0x7f251006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:34.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.813+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f251c0abf50 con 0x7f252c0d1000 2024-03-24T10:34:34.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.813+0000 7f2532057640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 0x7f251006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:34.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.813+0000 7f2532057640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 0x7f251006dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2524009a20 tx=0x7f2524009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:34.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:34.817+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f252c0b7900 con 0x7f252c0d1000 2024-03-24T10:34:35.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.013+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f252c0b7900 con 0x7f252c0d1000 2024-03-24T10:34:35.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.014+0000 7f2522ffd640 1 -- 172.21.15.1:0/4183897588 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f252c0b7900 con 0x7f252c0d1000 2024-03-24T10:34:35.014 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 msgr2=0x7f251006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 0x7f251006dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2524009a20 tx=0x7f2524009190 comp rx=0 tx=0).stop 2024-03-24T10:34:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 msgr2=0x7f252c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f251c02b830 tx=0x7f251c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 shutdown_connections 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f251006b810 0x7f251006dcd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f252c15e390 0x7f252c09f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f252c0d1000 0x7f252c0b7be0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 --2- 172.21.15.1:0/4183897588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f252c0b3320 0x7f252c155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 >> 172.21.15.1:0/4183897588 conn(0x7f252c0d3ec0 msgr2=0x7f252c0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 shutdown_connections 2024-03-24T10:34:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.016+0000 7f25342bb640 1 -- 172.21.15.1:0/4183897588 wait complete. 2024-03-24T10:34:35.035 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:34:35.035 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:35.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.193+0000 7f2bb0c33640 1 Processor -- start 2024-03-24T10:34:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.193+0000 7f2bb0c33640 1 -- start start 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2bb0c33640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2bac15e390 0x7f2bac0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2bb0c33640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bac15e770 con 0x7f2bac0b4ba0 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2bb0c33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bac0d4e40 con 0x7f2bac0d1000 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2bb0c33640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2bac0b3190 con 0x7f2bac15e390 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2baa59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2bac15e390 0x7f2bac0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2baad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2bac0b4ba0 0x7f2bac0cf170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52830/0 (socket says 172.21.15.1:52830) 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.194+0000 7f2baad9d640 1 -- 172.21.15.1:0/3228481245 learned_addr learned my addr 172.21.15.1:0/3228481245 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.195+0000 7f2baa59c640 1 -- 172.21.15.1:0/3228481245 >> v1:172.21.15.3:6789/0 conn(0x7f2bac0d1000 legacy=0x7f2bac0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.195+0000 7f2baa59c640 1 -- 172.21.15.1:0/3228481245 >> v1:172.21.15.1:6789/0 conn(0x7f2bac0b4ba0 legacy=0x7f2bac0cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.195+0000 7f2ba959a640 1 -- 172.21.15.1:0/3228481245 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3552924040 0 0) 0x7f2bac0d4e40 con 0x7f2bac0d1000 2024-03-24T10:34:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.195+0000 7f2baa59c640 1 -- 172.21.15.1:0/3228481245 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bac0d1e40 con 0x7f2bac15e390 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.195+0000 7f2baa59c640 1 --2- 172.21.15.1:0/3228481245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac15e390 0x7f2bac0d24a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009d90 tx=0x7f2b9c02bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d98c2af99f3bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2ba959a640 1 -- 172.21.15.1:0/3228481245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c02c990 con 0x7f2bac15e390 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2ba959a640 1 -- 172.21.15.1:0/3228481245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b9c02caf0 con 0x7f2bac15e390 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2ba959a640 1 -- 172.21.15.1:0/3228481245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c02ce80 con 0x7f2bac15e390 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 -- 172.21.15.1:0/3228481245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac15e390 msgr2=0x7f2bac0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/3228481245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac15e390 0x7f2bac0d24a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009d90 tx=0x7f2b9c02bc60 comp rx=0 tx=0).stop 2024-03-24T10:34:35.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 -- 172.21.15.1:0/3228481245 shutdown_connections 2024-03-24T10:34:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/3228481245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac15e390 0x7f2bac0d24a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 -- 172.21.15.1:0/3228481245 >> 172.21.15.1:0/3228481245 conn(0x7f2bac0d3ec0 msgr2=0x7f2bac0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 -- 172.21.15.1:0/3228481245 shutdown_connections 2024-03-24T10:34:35.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.196+0000 7f2bb0c33640 1 -- 172.21.15.1:0/3228481245 wait complete. 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 Processor -- start 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 -- start start 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bac0b4ba0 0x7f2bac14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2bac15e390 0x7f2bac0b9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac14ff00 con 0x7f2bac0d1000 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac152b40 con 0x7f2bac0b4ba0 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2bb0c33640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2bac1501c0 con 0x7f2bac15e390 2024-03-24T10:34:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2ba9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2ba9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43512/0 (socket says 172.21.15.1:43512) 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2ba9d9b640 1 -- 172.21.15.1:0/85061306 learned_addr learned my addr 172.21.15.1:0/85061306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2baa59c640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bac0b4ba0 0x7f2bac14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.198+0000 7f2baad9d640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2bac15e390 0x7f2bac0b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 -- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2bac15e390 msgr2=0x7f2bac0b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2bac15e390 0x7f2bac0b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 -- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bac0b4ba0 msgr2=0x7f2bac14f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bac0b4ba0 0x7f2bac14f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 -- 172.21.15.1:0/85061306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9c009a70 con 0x7f2bac0d1000 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba9d9b640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400c3e0 tx=0x7f2b9400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b940035f0 con 0x7f2bac0d1000 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b94003750 con 0x7f2bac0d1000 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bac1504a0 con 0x7f2bac0d1000 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b94003b20 con 0x7f2bac0d1000 2024-03-24T10:34:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.199+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bac151db0 con 0x7f2bac0d1000 2024-03-24T10:34:35.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.200+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bac0d42a0 con 0x7f2bac0d1000 2024-03-24T10:34:35.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.200+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2b94003c80 con 0x7f2bac0d1000 2024-03-24T10:34:35.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.201+0000 7f2ba2ffd640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 0x7f2b8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.201+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b940896a0 con 0x7f2bac0d1000 2024-03-24T10:34:35.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.201+0000 7f2baa59c640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 0x7f2b8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.201+0000 7f2baa59c640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 0x7f2b8c06dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c036000 tx=0x7f2b9c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.205+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2b94054200 con 0x7f2bac0d1000 2024-03-24T10:34:35.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.394+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bac150690 con 0x7f2bac0d1000 2024-03-24T10:34:35.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.395+0000 7f2ba2ffd640 1 -- 172.21.15.1:0/85061306 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16832 (secure 0 0 0) 0x7f2b940578c0 con 0x7f2bac0d1000 2024-03-24T10:34:35.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:35.396 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":22,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:35.248842+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:35.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 msgr2=0x7f2b8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 0x7f2b8c06dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c036000 tx=0x7f2b9c030000 comp rx=0 tx=0).stop 2024-03-24T10:34:35.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 msgr2=0x7f2bac0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400c3e0 tx=0x7f2b9400c8a0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 shutdown_connections 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2b8c06b810 0x7f2b8c06dcd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2bac15e390 0x7f2bac0b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bac0d1000 0x7f2bac0b8af0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 --2- 172.21.15.1:0/85061306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bac0b4ba0 0x7f2bac14f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 >> 172.21.15.1:0/85061306 conn(0x7f2bac0d3ec0 msgr2=0x7f2bac0d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 shutdown_connections 2024-03-24T10:34:35.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.397+0000 7f2bb0c33640 1 -- 172.21.15.1:0/85061306 wait complete. 2024-03-24T10:34:35.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:35.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.512+0000 7fac551d0640 1 Processor -- start 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac551d0640 1 -- start start 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac551d0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac5006f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac551d0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac500cc2b0 con 0x7fac5006f780 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac551d0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac500cc870 con 0x7fac500cdd10 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac551d0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fac500cf460 con 0x7fac5015e2c0 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac47fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac5006f780 0x7fac500ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52836/0 (socket says 172.21.15.1:52836) 2024-03-24T10:34:35.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.513+0000 7fac47fff640 1 -- 172.21.15.1:0/680277477 learned_addr learned my addr 172.21.15.1:0/680277477 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4ed9d640 1 --2- 172.21.15.1:0/680277477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac5006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4173952378 0 0) 0x7fac500cc2b0 con 0x7fac5006f780 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac3c003520 con 0x7fac5006f780 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968435471 0 0) 0x7fac500cc870 con 0x7fac500cdd10 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac500cc2b0 con 0x7fac500cdd10 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4ed9d640 1 -- 172.21.15.1:0/680277477 >> v1:172.21.15.3:6789/0 conn(0x7fac500cdd10 legacy=0x7fac500ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4ed9d640 1 -- 172.21.15.1:0/680277477 >> v1:172.21.15.1:6789/0 conn(0x7fac5006f780 legacy=0x7fac500ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.514+0000 7fac4ed9d640 1 -- 172.21.15.1:0/680277477 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac500d0d50 con 0x7fac5015e2c0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac4ed9d640 1 --2- 172.21.15.1:0/680277477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac5006f130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac400097d0 tx=0x7fac4002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35011f9e6581f8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40031800 con 0x7fac5015e2c0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac40031960 con 0x7fac5015e2c0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac4e59c640 1 -- 172.21.15.1:0/680277477 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40031cf0 con 0x7fac5015e2c0 2024-03-24T10:34:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 -- 172.21.15.1:0/680277477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 msgr2=0x7fac5006f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 --2- 172.21.15.1:0/680277477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac5006f130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac400097d0 tx=0x7fac4002b8a0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 -- 172.21.15.1:0/680277477 shutdown_connections 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 --2- 172.21.15.1:0/680277477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac5006f130 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 -- 172.21.15.1:0/680277477 >> 172.21.15.1:0/680277477 conn(0x7fac500d1c60 msgr2=0x7fac500d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 -- 172.21.15.1:0/680277477 shutdown_connections 2024-03-24T10:34:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.515+0000 7fac551d0640 1 -- 172.21.15.1:0/680277477 wait complete. 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 Processor -- start 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 -- start start 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac500cdd10 0x7fac501f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac500b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fac501f8230 con 0x7fac5006f780 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fac500b5a10 con 0x7fac500cdd10 2024-03-24T10:34:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.517+0000 7fac551d0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fac500b5170 con 0x7fac5015e2c0 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43526/0 (socket says 172.21.15.1:43526) 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 -- 172.21.15.1:0/1458997525 learned_addr learned my addr 172.21.15.1:0/1458997525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac47fff640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac500cdd10 0x7fac501f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4f59e640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac500b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 -- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 msgr2=0x7fac500b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac500b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 -- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac500cdd10 msgr2=0x7fac501f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac500cdd10 0x7fac501f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 -- 172.21.15.1:0/1458997525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac40009400 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac4ed9d640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fac400098f0 tx=0x7fac400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4003bc70 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4003be10 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac500b5450 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40045630 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.518+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac500b5f70 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.519+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac50068680 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.520+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac4004e450 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.520+0000 7fac477fe640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 0x7fac3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.520+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5891+0+0 (secure 0 0 0) 0x7fac4002e070 con 0x7fac5006f780 2024-03-24T10:34:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.520+0000 7fac47fff640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 0x7fac3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.521+0000 7fac47fff640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 0x7fac3006dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fac501575c0 tx=0x7fac38007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.525+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac40076dd0 con 0x7fac5006f780 2024-03-24T10:34:35.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.704+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac5004f4e0 con 0x7fac5006f780 2024-03-24T10:34:35.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.704+0000 7fac477fe640 1 -- 172.21.15.1:0/1458997525 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fac40006e10 con 0x7fac5006f780 2024-03-24T10:34:35.707 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 msgr2=0x7fac3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 0x7fac3006dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fac501575c0 tx=0x7fac38007d70 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 msgr2=0x7fac500b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fac400098f0 tx=0x7fac400046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 shutdown_connections 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fac3006b810 0x7fac3006dcd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fac5015e2c0 0x7fac500b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac500cdd10 0x7fac501f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 --2- 172.21.15.1:0/1458997525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006f780 0x7fac500b4ab0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 >> 172.21.15.1:0/1458997525 conn(0x7fac500d1c60 msgr2=0x7fac500cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 shutdown_connections 2024-03-24T10:34:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.706+0000 7fac551d0640 1 -- 172.21.15.1:0/1458997525 wait complete. 2024-03-24T10:34:35.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 Processor -- start 2024-03-24T10:34:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 -- start start 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 0x7fbb340d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbb340b4e40 0x7fbb34161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb34162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbb340cfd50 con 0x7fbb340b4e40 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbb340c7a40 con 0x7fbb341615f0 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb38c4e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340b55b0 con 0x7fbb341619d0 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.822+0000 7fbb2bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbb340b4e40 0x7fbb34161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbb340b4e40 0x7fbb34161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43540/0 (socket says 172.21.15.1:43540) 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 -- 172.21.15.1:0/2440498215 learned_addr learned my addr 172.21.15.1:0/2440498215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb32d9d640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb34162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb3259c640 1 --2- 172.21.15.1:0/2440498215 >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 0x7fbb340d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 -- 172.21.15.1:0/2440498215 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 msgr2=0x7fbb34162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:35.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb34162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 -- 172.21.15.1:0/2440498215 >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 msgr2=0x7fbb340d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/2440498215 >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 0x7fbb340d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb2bfff640 1 -- 172.21.15.1:0/2440498215 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb340b52d0 con 0x7fbb340b4e40 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.823+0000 7fbb3259c640 1 --2- 172.21.15.1:0/2440498215 >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 0x7fbb340d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb340b4e40 0x7fbb34161f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbb340b69f0 tx=0x7fbb1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0a65bf6b17b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb31d9b640 1 -- 172.21.15.1:0/2440498215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c031640 con 0x7fbb340b4e40 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb31d9b640 1 -- 172.21.15.1:0/2440498215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1c0317a0 con 0x7fbb340b4e40 2024-03-24T10:34:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb31d9b640 1 -- 172.21.15.1:0/2440498215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c031b30 con 0x7fbb340b4e40 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 -- 172.21.15.1:0/2440498215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb340b4e40 msgr2=0x7fbb34161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb340b4e40 0x7fbb34161f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbb340b69f0 tx=0x7fbb1c02b8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 -- 172.21.15.1:0/2440498215 shutdown_connections 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb34162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/2440498215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb340b4e40 0x7fbb34161f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/2440498215 >> v2:172.21.15.3:3300/0 conn(0x7fbb341615f0 0x7fbb340d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 -- 172.21.15.1:0/2440498215 >> 172.21.15.1:0/2440498215 conn(0x7fbb340d3ef0 msgr2=0x7fbb340d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 -- 172.21.15.1:0/2440498215 shutdown_connections 2024-03-24T10:34:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.824+0000 7fbb38c4e640 1 -- 172.21.15.1:0/2440498215 wait complete. 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.825+0000 7fbb38c4e640 1 Processor -- start 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 -- start start 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb3415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb3415b2b0 0x7fbb340c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb340cbce0 con 0x7fbb341615f0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3415b760 con 0x7fbb3415b2b0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb38c4e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340cb060 con 0x7fbb341619d0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb3259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb3259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43552/0 (socket says 172.21.15.1:43552) 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb3259c640 1 -- 172.21.15.1:0/311584331 learned_addr learned my addr 172.21.15.1:0/311584331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb32d9d640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb3415b2b0 0x7fbb340c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.826+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb3415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 -- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 msgr2=0x7fbb3415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb3415ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 -- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb3415b2b0 msgr2=0x7fbb340c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb3415b2b0 0x7fbb340c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 -- 172.21.15.1:0/311584331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb24009400 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb3259c640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e980 tx=0x7fbb2400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240096e0 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2400ca00 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400cd90 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c0094d0 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.827+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb340cb620 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.828+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3404f470 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.828+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb24013470 con 0x7fbb341615f0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.829+0000 7fbb2affd640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 0x7fbb1406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.829+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 0x7fbb1406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:35.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.829+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5891+0+0 (secure 0 0 0) 0x7fbb2408a470 con 0x7fbb341615f0 2024-03-24T10:34:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.829+0000 7fbb2bfff640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 0x7fbb1406dd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:35.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:35.833+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb24054df0 con 0x7fbb341615f0 2024-03-24T10:34:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.013+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbb3408d640 con 0x7fbb341615f0 2024-03-24T10:34:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.013+0000 7fbb2affd640 1 -- 172.21.15.1:0/311584331 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fbb240584b0 con 0x7fbb341615f0 2024-03-24T10:34:36.013 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 msgr2=0x7fbb1406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 0x7fbb1406dd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 msgr2=0x7fbb34158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e980 tx=0x7fbb2400ee40 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 shutdown_connections 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbb1406b840 0x7fbb1406dd00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb3415b2b0 0x7fbb340c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbb341619d0 0x7fbb3415ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 --2- 172.21.15.1:0/311584331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb341615f0 0x7fbb34158e70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 >> 172.21.15.1:0/311584331 conn(0x7fbb340d3ef0 msgr2=0x7fbb340d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 shutdown_connections 2024-03-24T10:34:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.015+0000 7fbb38c4e640 1 -- 172.21.15.1:0/311584331 wait complete. 2024-03-24T10:34:36.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.130+0000 7f6047d93640 1 Processor -- start 2024-03-24T10:34:36.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6047d93640 1 -- start start 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6047d93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 0x7f60400d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6047d93640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60400b0b70 con 0x7f60400cfae0 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6047d93640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60400d1d60 con 0x7f60400b2ed0 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6047d93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60400b0de0 con 0x7f604015e390 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.131+0000 7f6046330640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60400cfae0 0x7f60400b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52840/0 (socket says 172.21.15.1:52840) 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6046330640 1 -- 172.21.15.1:0/2190226079 learned_addr learned my addr 172.21.15.1:0/2190226079 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6045b2f640 1 --2- 172.21.15.1:0/2190226079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 0x7f60400d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3706536429 0 0) 0x7f60400d1d60 con 0x7f60400b2ed0 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f602c003520 con 0x7f60400b2ed0 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193217663 0 0) 0x7f60400b0b70 con 0x7f60400cfae0 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60400d1d60 con 0x7f60400cfae0 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6045b2f640 1 -- 172.21.15.1:0/2190226079 >> v1:172.21.15.3:6789/0 conn(0x7f60400b2ed0 legacy=0x7f60400b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6045b2f640 1 -- 172.21.15.1:0/2190226079 >> v1:172.21.15.1:6789/0 conn(0x7f60400cfae0 legacy=0x7f60400b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.132+0000 7f6045b2f640 1 -- 172.21.15.1:0/2190226079 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604015e770 con 0x7f604015e390 2024-03-24T10:34:36.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6045b2f640 1 --2- 172.21.15.1:0/2190226079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 0x7f60400d2310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f603c0097d0 tx=0x7f603c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20c938aff22cd628 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c031890 con 0x7f604015e390 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c0319f0 con 0x7f604015e390 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6044b2d640 1 -- 172.21.15.1:0/2190226079 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c031d80 con 0x7f604015e390 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 -- 172.21.15.1:0/2190226079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 msgr2=0x7f60400d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 --2- 172.21.15.1:0/2190226079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 0x7f60400d2310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f603c0097d0 tx=0x7f603c02b930 comp rx=0 tx=0).stop 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 -- 172.21.15.1:0/2190226079 shutdown_connections 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 --2- 172.21.15.1:0/2190226079 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f604015e390 0x7f60400d2310 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 -- 172.21.15.1:0/2190226079 >> 172.21.15.1:0/2190226079 conn(0x7f60400d3ee0 msgr2=0x7f60400d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 -- 172.21.15.1:0/2190226079 shutdown_connections 2024-03-24T10:34:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.133+0000 7f6047d93640 1 -- 172.21.15.1:0/2190226079 wait complete. 2024-03-24T10:34:36.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 Processor -- start 2024-03-24T10:34:36.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 -- start start 2024-03-24T10:34:36.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 0x7f60401586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60400cfae0 0x7f60401f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f60400b6b60 con 0x7f604015e390 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f60401f7b70 con 0x7f60400b2ed0 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6047d93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60401f7e50 con 0x7f60400cfae0 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6046330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6045b2f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 0x7f60401586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6046330640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43564/0 (socket says 172.21.15.1:43564) 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f6046330640 1 -- 172.21.15.1:0/3417824715 learned_addr learned my addr 172.21.15.1:0/3417824715 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.135+0000 7f604532e640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60400cfae0 0x7f60401f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 -- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60400cfae0 msgr2=0x7f60401f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60400cfae0 0x7f60401f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 -- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 msgr2=0x7f60401586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 0x7f60401586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 -- 172.21.15.1:0/3417824715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603c009400 con 0x7f604015e390 2024-03-24T10:34:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6045b2f640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 0x7f60401586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6046330640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f603800cb40 tx=0x7f603800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603800aca0 con 0x7f604015e390 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603800ae00 con 0x7f604015e390 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60400b6dd0 con 0x7f604015e390 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6038012620 con 0x7f604015e390 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.136+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60401f8130 con 0x7f604015e390 2024-03-24T10:34:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.137+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604004f4e0 con 0x7f604015e390 2024-03-24T10:34:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.138+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6038007070 con 0x7f604015e390 2024-03-24T10:34:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.138+0000 7f60367fc640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 0x7f602406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.138+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5891+0+0 (secure 0 0 0) 0x7f603808a2f0 con 0x7f604015e390 2024-03-24T10:34:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.138+0000 7f6045b2f640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 0x7f602406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.139+0000 7f6045b2f640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 0x7f602406dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f603c036000 tx=0x7f603c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.143+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6038054c70 con 0x7f604015e390 2024-03-24T10:34:36.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.327+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60400d3210 con 0x7f604015e390 2024-03-24T10:34:36.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.328+0000 7f60367fc640 1 -- 172.21.15.1:0/3417824715 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f6038058330 con 0x7f604015e390 2024-03-24T10:34:36.328 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:36.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.329+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 msgr2=0x7f602406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.329+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 0x7f602406dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f603c036000 tx=0x7f603c03c040 comp rx=0 tx=0).stop 2024-03-24T10:34:36.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.329+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 msgr2=0x7f60400ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.329+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f603800cb40 tx=0x7f603800a060 comp rx=0 tx=0).stop 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 shutdown_connections 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f602406b810 0x7f602406dcd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f604015e390 0x7f60400ccd30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60400cfae0 0x7f60401f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 --2- 172.21.15.1:0/3417824715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60400b2ed0 0x7f60401586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 >> 172.21.15.1:0/3417824715 conn(0x7f60400d3ee0 msgr2=0x7f60400b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 shutdown_connections 2024-03-24T10:34:36.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.330+0000 7f6047d93640 1 -- 172.21.15.1:0/3417824715 wait complete. 2024-03-24T10:34:36.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.443+0000 7effe362e640 1 Processor -- start 2024-03-24T10:34:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.443+0000 7effe362e640 1 -- start start 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe362e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effdc15e0a0 0x7effdc06b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe362e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effdc162a90 con 0x7effdc072710 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe362e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effdc072c10 con 0x7effdc0b0180 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe362e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7effdc06dde0 con 0x7effdc15e0a0 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe13ca640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effdc15e0a0 0x7effdc06b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe0bc9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7effdc072710 0x7effdc161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52844/0 (socket says 172.21.15.1:52844) 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe0bc9640 1 -- 172.21.15.1:0/2170493911 learned_addr learned my addr 172.21.15.1:0/2170493911 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe13ca640 1 -- 172.21.15.1:0/2170493911 >> v1:172.21.15.3:6789/0 conn(0x7effdc0b0180 legacy=0x7effdc162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.444+0000 7effe13ca640 1 -- 172.21.15.1:0/2170493911 >> v1:172.21.15.1:6789/0 conn(0x7effdc072710 legacy=0x7effdc161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effdbfff640 1 -- 172.21.15.1:0/2170493911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272941306 0 0) 0x7effdc162a90 con 0x7effdc072710 2024-03-24T10:34:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effe13ca640 1 -- 172.21.15.1:0/2170493911 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effdc06e0c0 con 0x7effdc15e0a0 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effe13ca640 1 --2- 172.21.15.1:0/2170493911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc15e0a0 0x7effdc06b0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7effd40097a0 tx=0x7effd402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e11b266df944ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effdbfff640 1 -- 172.21.15.1:0/2170493911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4030610 con 0x7effdc15e0a0 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effdbfff640 1 -- 172.21.15.1:0/2170493911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd4030770 con 0x7effdc15e0a0 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.445+0000 7effdbfff640 1 -- 172.21.15.1:0/2170493911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4030b00 con 0x7effdc15e0a0 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 -- 172.21.15.1:0/2170493911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc15e0a0 msgr2=0x7effdc06b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 --2- 172.21.15.1:0/2170493911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc15e0a0 0x7effdc06b0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7effd40097a0 tx=0x7effd402b960 comp rx=0 tx=0).stop 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 -- 172.21.15.1:0/2170493911 shutdown_connections 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 --2- 172.21.15.1:0/2170493911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc15e0a0 0x7effdc06b0a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 -- 172.21.15.1:0/2170493911 >> 172.21.15.1:0/2170493911 conn(0x7effdc0b2820 msgr2=0x7effdc0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 -- 172.21.15.1:0/2170493911 shutdown_connections 2024-03-24T10:34:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.446+0000 7effe362e640 1 -- 172.21.15.1:0/2170493911 wait complete. 2024-03-24T10:34:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.447+0000 7effe362e640 1 Processor -- start 2024-03-24T10:34:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.447+0000 7effe362e640 1 -- start start 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effdc0b0180 0x7effdc0cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effdc15e0a0 0x7effdc1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7effdc14e3e0 con 0x7effdc072710 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7effdc150b40 con 0x7effdc0b0180 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe362e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7effdc150e20 con 0x7effdc15e0a0 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43590/0 (socket says 172.21.15.1:43590) 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 -- 172.21.15.1:0/3504380280 learned_addr learned my addr 172.21.15.1:0/3504380280 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe0bc9640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effdc0b0180 0x7effdc0cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe1bcb640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effdc15e0a0 0x7effdc1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 -- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effdc15e0a0 msgr2=0x7effdc1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effdc15e0a0 0x7effdc1517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 -- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effdc0b0180 msgr2=0x7effdc0cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effdc0b0180 0x7effdc0cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.448+0000 7effe13ca640 1 -- 172.21.15.1:0/3504380280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd4009400 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effe13ca640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7effd402bf70 tx=0x7effd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd403e040 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd4039cb0 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effdc0cd740 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4044630 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.449+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effdc156ef0 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.450+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effdc0cd5a0 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.450+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7effd404d3b0 con 0x7effdc072710 2024-03-24T10:34:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.451+0000 7effd9ffb640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 0x7effc806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.451+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7effd40ac730 con 0x7effdc072710 2024-03-24T10:34:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.451+0000 7effe0bc9640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 0x7effc806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.451+0000 7effe0bc9640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 0x7effc806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7effdc0ccc20 tx=0x7effd0007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.455+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7effdc0cd5a0 con 0x7effdc072710 2024-03-24T10:34:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.651+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7effdc0cdc60 con 0x7effdc072710 2024-03-24T10:34:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.652+0000 7effd9ffb640 1 -- 172.21.15.1:0/3504380280 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7effdc0cdc60 con 0x7effdc072710 2024-03-24T10:34:36.652 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 msgr2=0x7effc806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 0x7effc806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7effdc0ccc20 tx=0x7effd0007e40 comp rx=0 tx=0).stop 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 msgr2=0x7effdc14dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7effd402bf70 tx=0x7effd4004730 comp rx=0 tx=0).stop 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 shutdown_connections 2024-03-24T10:34:36.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effc806b840 0x7effc806dd00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effdc15e0a0 0x7effdc1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effdc0b0180 0x7effdc0cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 --2- 172.21.15.1:0/3504380280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc072710 0x7effdc14dea0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 >> 172.21.15.1:0/3504380280 conn(0x7effdc0b2820 msgr2=0x7effdc0ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 shutdown_connections 2024-03-24T10:34:36.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.654+0000 7effe362e640 1 -- 172.21.15.1:0/3504380280 wait complete. 2024-03-24T10:34:36.667 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:34:36.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:36.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-24T10:34:36.694 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:36.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:34:36.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:36.695 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T10:34:36.750 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:36.750 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:36.769 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:36.769 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:36.825 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 Processor -- start 2024-03-24T10:34:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 -- start start 2024-03-24T10:34:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e7c070b90 0x7f4e7c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c0cdd60 con 0x7f4e7c0b2aa0 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c06eaa0 con 0x7f4e7c070b90 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.927+0000 7f4e82b5c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c071010 con 0x7f4e7c144770 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e808f8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e7c070b90 0x7f4e7c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e810f9640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4e7c0b2aa0 0x7f4e7c0ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:34104/0 (socket says 172.21.15.1:34104) 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e808f8640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:45732/0 (socket says 172.21.15.1:45732) 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e810f9640 1 -- 172.21.15.1:0/4010952386 learned_addr learned my addr 172.21.15.1:0/4010952386 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e7c070b90 0x7f4e7c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:48990/0 (socket says 172.21.15.1:48990) 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 -- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 msgr2=0x7f4e7c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 -- 172.21.15.1:0/4010952386 >> v1:172.21.15.3:6789/0 conn(0x7f4e7c0b2aa0 legacy=0x7f4e7c0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7bfff640 1 -- 172.21.15.1:0/4010952386 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c06df60 con 0x7f4e7c070b90 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e7b7fe640 1 -- 172.21.15.1:0/4010952386 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216881565 0 0) 0x7f4e7c0cdd60 con 0x7f4e7c0b2aa0 2024-03-24T10:34:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.928+0000 7f4e808f8640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e7bfff640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c06da20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4e680097a0 tx=0x7f4e6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4536622855f42527 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e7b7fe640 1 -- 172.21.15.1:0/4010952386 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68031830 con 0x7f4e7c070b90 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e7b7fe640 1 -- 172.21.15.1:0/4010952386 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68031990 con 0x7f4e7c070b90 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e7b7fe640 1 -- 172.21.15.1:0/4010952386 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68031d20 con 0x7f4e7c070b90 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 -- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 msgr2=0x7f4e7c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c06da20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4e680097a0 tx=0x7f4e6802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 -- 172.21.15.1:0/4010952386 shutdown_connections 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c06da20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/4010952386 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c144770 0x7f4e7c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.929+0000 7f4e82b5c640 1 -- 172.21.15.1:0/4010952386 >> 172.21.15.1:0/4010952386 conn(0x7f4e7c0b17a0 msgr2=0x7f4e7c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.930+0000 7f4e82b5c640 1 -- 172.21.15.1:0/4010952386 shutdown_connections 2024-03-24T10:34:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.930+0000 7f4e82b5c640 1 -- 172.21.15.1:0/4010952386 wait complete. 2024-03-24T10:34:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.931+0000 7f4e82b5c640 1 Processor -- start 2024-03-24T10:34:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.931+0000 7f4e82b5c640 1 -- start start 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.931+0000 7f4e82b5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c1f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.931+0000 7f4e82b5c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 0x7f4e7c13e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c140970 con 0x7f4e7c144770 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c139700 con 0x7f4e7c070b90 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c1f4bd0 con 0x7f4e7c0b2aa0 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e7bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e810f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 0x7f4e7c13e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e7bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:45734/0 (socket says 172.21.15.1:45734) 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c1f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e7bfff640 1 -- 172.21.15.1:0/192010857 learned_addr learned my addr 172.21.15.1:0/192010857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 -- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 msgr2=0x7f4e7c1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 -- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 msgr2=0x7f4e7c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 0x7f4e7c13e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 -- 172.21.15.1:0/192010857 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e70009400 con 0x7f4e7c070b90 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e7bfff640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e810f9640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 0x7f4e7c13e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e808f8640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c1f4690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e870 tx=0x7f4e7000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000c940 con 0x7f4e7c070b90 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7000caa0 con 0x7f4e7c070b90 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e68009400 con 0x7f4e7c070b90 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000ce30 con 0x7f4e7c070b90 2024-03-24T10:34:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.932+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7c140740 con 0x7f4e7c070b90 2024-03-24T10:34:36.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.933+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7c139840 con 0x7f4e7c070b90 2024-03-24T10:34:36.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.934+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e70013470 con 0x7f4e7c070b90 2024-03-24T10:34:36.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.934+0000 7f4e797fa640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 0x7f4e6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:36.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.934+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f4e7008a2f0 con 0x7f4e7c070b90 2024-03-24T10:34:36.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.934+0000 7f4e7bfff640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 0x7f4e6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.935+0000 7f4e7bfff640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 0x7f4e6006dcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0af2c0 tx=0x7f4e6803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:36.939+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e70054d90 con 0x7f4e7c070b90 2024-03-24T10:34:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.119+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e7c04f4e0 con 0x7f4e7c070b90 2024-03-24T10:34:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.120+0000 7f4e797fa640 1 -- 172.21.15.1:0/192010857 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16832 (secure 0 0 0) 0x7f4e70058450 con 0x7f4e7c070b90 2024-03-24T10:34:37.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:37.122 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:36.249848+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 msgr2=0x7f4e6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 0x7f4e6006dcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0af2c0 tx=0x7f4e6803c040 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 msgr2=0x7f4e7c1f4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c1f4690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e870 tx=0x7f4e7000ed30 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 shutdown_connections 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e6006b810 0x7f4e6006dcd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e7c144770 0x7f4e7c13e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e7c0b2aa0 0x7f4e7c1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 --2- 172.21.15.1:0/192010857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e7c070b90 0x7f4e7c1f4690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 >> 172.21.15.1:0/192010857 conn(0x7f4e7c0b17a0 msgr2=0x7f4e7c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 shutdown_connections 2024-03-24T10:34:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.122+0000 7f4e82b5c640 1 -- 172.21.15.1:0/192010857 wait complete. 2024-03-24T10:34:37.133 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:37.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.238+0000 7fb146b80640 1 Processor -- start 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.238+0000 7fb146b80640 1 -- start start 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb146b80640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb14006fe90 0x7fb14006f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb146b80640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb1400cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb146b80640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1400cc2b0 con 0x7fb14015e2c0 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb146b80640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb1400cc870 con 0x7fb14006fe90 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb146b80640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb1400d0d50 con 0x7fb1400ccd00 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb14006fe90 0x7fb14006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb14491c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb14015e2c0 0x7fb14006f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:34116/0 (socket says 172.21.15.1:34116) 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb14491c640 1 -- 172.21.15.1:0/2215772988 learned_addr learned my addr 172.21.15.1:0/2215772988 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb14006fe90 0x7fb14006f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49008/0 (socket says 172.21.15.1:49008) 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb14511d640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb1400cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 -- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 msgr2=0x7fb1400cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb1400cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 -- 172.21.15.1:0/2215772988 >> v1:172.21.15.3:6789/0 conn(0x7fb14015e2c0 legacy=0x7fb14006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13f7fe640 1 -- 172.21.15.1:0/2215772988 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911408953 0 0) 0x7fb1400cc2b0 con 0x7fb14015e2c0 2024-03-24T10:34:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.239+0000 7fb13ffff640 1 -- 172.21.15.1:0/2215772988 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1400d1030 con 0x7fb14006fe90 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.240+0000 7fb13ffff640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb14006f130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0097a0 tx=0x7fb12c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3019754eee640a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.240+0000 7fb13f7fe640 1 -- 172.21.15.1:0/2215772988 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02e070 con 0x7fb14006fe90 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.240+0000 7fb13f7fe640 1 -- 172.21.15.1:0/2215772988 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12c0315a0 con 0x7fb14006fe90 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.240+0000 7fb13f7fe640 1 -- 172.21.15.1:0/2215772988 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c031930 con 0x7fb14006fe90 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 -- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 msgr2=0x7fb14006f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb14006f130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0097a0 tx=0x7fb12c02b840 comp rx=0 tx=0).stop 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 -- 172.21.15.1:0/2215772988 shutdown_connections 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb1400cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 --2- 172.21.15.1:0/2215772988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb14006f130 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 -- 172.21.15.1:0/2215772988 >> 172.21.15.1:0/2215772988 conn(0x7fb1400d1c60 msgr2=0x7fb1400d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 -- 172.21.15.1:0/2215772988 shutdown_connections 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.241+0000 7fb146b80640 1 -- 172.21.15.1:0/2215772988 wait complete. 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.242+0000 7fb146b80640 1 Processor -- start 2024-03-24T10:34:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.242+0000 7fb146b80640 1 -- start start 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb140160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb14014bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400b4620 con 0x7fb14015e2c0 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb140160800 con 0x7fb14006fe90 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb146b80640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb1401499b0 con 0x7fb1400ccd00 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14491c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb140160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43602/0 (socket says 172.21.15.1:43602) 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 -- 172.21.15.1:0/2570823433 learned_addr learned my addr 172.21.15.1:0/2570823433 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:37.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb13ffff640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb14014bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 -- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 msgr2=0x7fb14014bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb14014bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 -- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 msgr2=0x7fb140160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb140160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 -- 172.21.15.1:0/2570823433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c009400 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14491c640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb140160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.243+0000 7fb14511d640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb13800efc0 tx=0x7fb13800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.244+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138014480 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.244+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13800f040 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.244+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138014820 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.244+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14014cb60 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.244+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1400b4060 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.245+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb140160940 con 0x7fb14015e2c0 2024-03-24T10:34:37.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.245+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb138014980 con 0x7fb14015e2c0 2024-03-24T10:34:37.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.246+0000 7fb13d7fa640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 0x7fb12406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.246+0000 7fb14491c640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 0x7fb12406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.246+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fb13808ace0 con 0x7fb14015e2c0 2024-03-24T10:34:37.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.246+0000 7fb14491c640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 0x7fb12406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb134007fd0 tx=0x7fb134007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.250+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1380556f0 con 0x7fb14015e2c0 2024-03-24T10:34:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.430+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb14004f4e0 con 0x7fb14015e2c0 2024-03-24T10:34:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.430+0000 7fb13d7fa640 1 -- 172.21.15.1:0/2570823433 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fb138018020 con 0x7fb14015e2c0 2024-03-24T10:34:37.431 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 msgr2=0x7fb12406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 0x7fb12406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb134007fd0 tx=0x7fb134007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 msgr2=0x7fb14014c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb13800efc0 tx=0x7fb13800c490 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 shutdown_connections 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb12406b810 0x7fb12406dcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb14015e2c0 0x7fb14014c1f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb1400ccd00 0x7fb14014bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 --2- 172.21.15.1:0/2570823433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb14006fe90 0x7fb140160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 >> 172.21.15.1:0/2570823433 conn(0x7fb1400d1c60 msgr2=0x7fb14006efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 shutdown_connections 2024-03-24T10:34:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.432+0000 7fb146b80640 1 -- 172.21.15.1:0/2570823433 wait complete. 2024-03-24T10:34:37.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:37.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.548+0000 7f5174c6f640 1 Processor -- start 2024-03-24T10:34:37.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.548+0000 7f5174c6f640 1 -- start start 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f5174c6f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51700b31d0 0x7f51700d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f5174c6f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f51700ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f5174c6f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51700cc250 con 0x7f51700d0e90 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f5174c6f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f51700b35b0 con 0x7f51700b31d0 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f5174c6f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f51700cc4a0 con 0x7f517006d240 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51700b31d0 0x7f51700d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51700b31d0 0x7f51700d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43614/0 (socket says 172.21.15.1:43614) 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 -- 172.21.15.1:0/2972283604 learned_addr learned my addr 172.21.15.1:0/2972283604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516dd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f51700d0e90 0x7f517006cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52856/0 (socket says 172.21.15.1:52856) 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516ed9d640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f51700ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 -- 172.21.15.1:0/2972283604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 msgr2=0x7f51700ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f51700ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 -- 172.21.15.1:0/2972283604 >> v1:172.21.15.1:6789/0 conn(0x7f51700d0e90 legacy=0x7f517006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516e59c640 1 -- 172.21.15.1:0/2972283604 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51700cc780 con 0x7f51700b31d0 2024-03-24T10:34:37.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.549+0000 7f516d59a640 1 -- 172.21.15.1:0/2972283604 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780539318 0 0) 0x7f51700cc250 con 0x7f51700d0e90 2024-03-24T10:34:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.550+0000 7f516e59c640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700b31d0 0x7f51700d0950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5160009e50 tx=0x7f516002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb1e658f450690 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.550+0000 7f516d59a640 1 -- 172.21.15.1:0/2972283604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516002cb00 con 0x7f51700b31d0 2024-03-24T10:34:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.550+0000 7f516d59a640 1 -- 172.21.15.1:0/2972283604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516002cc60 con 0x7f51700b31d0 2024-03-24T10:34:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.550+0000 7f516d59a640 1 -- 172.21.15.1:0/2972283604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600343f0 con 0x7f51700b31d0 2024-03-24T10:34:37.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 -- 172.21.15.1:0/2972283604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700b31d0 msgr2=0x7f51700d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700b31d0 0x7f51700d0950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5160009e50 tx=0x7f516002bf60 comp rx=0 tx=0).stop 2024-03-24T10:34:37.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 -- 172.21.15.1:0/2972283604 shutdown_connections 2024-03-24T10:34:37.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f51700ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 --2- 172.21.15.1:0/2972283604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700b31d0 0x7f51700d0950 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 -- 172.21.15.1:0/2972283604 >> 172.21.15.1:0/2972283604 conn(0x7f51700d1c00 msgr2=0x7f51700d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 -- 172.21.15.1:0/2972283604 shutdown_connections 2024-03-24T10:34:37.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.551+0000 7f5174c6f640 1 -- 172.21.15.1:0/2972283604 wait complete. 2024-03-24T10:34:37.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.552+0000 7f5174c6f640 1 Processor -- start 2024-03-24T10:34:37.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.552+0000 7f5174c6f640 1 -- start start 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f5170157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51700b31d0 0x7f51700c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51700ca3e0 con 0x7f51700d0e90 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5170158380 con 0x7f51700b31d0 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f5174c6f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f51700cb2f0 con 0x7f517006d240 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43624/0 (socket says 172.21.15.1:43624) 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 -- 172.21.15.1:0/4126077853 learned_addr learned my addr 172.21.15.1:0/4126077853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:37.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516dd9b640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51700b31d0 0x7f51700c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516e59c640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f5170157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 -- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 msgr2=0x7f5170157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f5170157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 -- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51700b31d0 msgr2=0x7f51700c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51700b31d0 0x7f51700c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.553+0000 7f516ed9d640 1 -- 172.21.15.1:0/4126077853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5160009b30 con 0x7f51700d0e90 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f516ed9d640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f516800e930 tx=0x7f516800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168009690 con 0x7f51700d0e90 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516800c980 con 0x7f51700d0e90 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51700c6c80 con 0x7f51700d0e90 2024-03-24T10:34:37.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516800cd30 con 0x7f51700d0e90 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.554+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51700c7240 con 0x7f51700d0e90 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.555+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517004f470 con 0x7f51700d0e90 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.555+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5168013470 con 0x7f51700d0e90 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.556+0000 7f5166ffd640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 0x7f515006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.556+0000 7f516e59c640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 0x7f515006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.556+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f516808a420 con 0x7f51700d0e90 2024-03-24T10:34:37.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.556+0000 7f516e59c640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 0x7f515006dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5160035000 tx=0x7f5160030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.560+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5168054ef0 con 0x7f51700d0e90 2024-03-24T10:34:37.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.740+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f51700b7a80 con 0x7f51700d0e90 2024-03-24T10:34:37.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.741+0000 7f5166ffd640 1 -- 172.21.15.1:0/4126077853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f51680585b0 con 0x7f51700d0e90 2024-03-24T10:34:37.741 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 msgr2=0x7f515006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 0x7f515006dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5160035000 tx=0x7f5160030000 comp rx=0 tx=0).stop 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 msgr2=0x7f51700ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f516800e930 tx=0x7f516800edf0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 shutdown_connections 2024-03-24T10:34:37.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f515006b810 0x7f515006dcd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51700d0e90 0x7f51700ca780 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51700b31d0 0x7f51700c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 --2- 172.21.15.1:0/4126077853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f517006d240 0x7f5170157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.742+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 >> 172.21.15.1:0/4126077853 conn(0x7f51700d1c00 msgr2=0x7f517006eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.743+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 shutdown_connections 2024-03-24T10:34:37.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.743+0000 7f5174c6f640 1 -- 172.21.15.1:0/4126077853 wait complete. 2024-03-24T10:34:37.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.857+0000 7fc23ddd7640 1 Processor -- start 2024-03-24T10:34:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc23ddd7640 1 -- start start 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc23ddd7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc23815c4d0 0x7fc23815c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc23ddd7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2380b34d0 con 0x7fc23806e5b0 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc23ddd7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2380b37b0 con 0x7fc2380b4570 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc23ddd7640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc2380b3a90 con 0x7fc23815c4d0 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc2377fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc23806e5b0 0x7fc23806e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52872/0 (socket says 172.21.15.1:52872) 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.858+0000 7fc2377fe640 1 -- 172.21.15.1:0/2961664551 learned_addr learned my addr 172.21.15.1:0/2961664551 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc237fff640 1 --2- 172.21.15.1:0/2961664551 >> v2:172.21.15.3:3300/0 conn(0x7fc23815c4d0 0x7fc23815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc2367fc640 1 -- 172.21.15.1:0/2961664551 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4154861330 0 0) 0x7fc2380b34d0 con 0x7fc23806e5b0 2024-03-24T10:34:37.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc2367fc640 1 -- 172.21.15.1:0/2961664551 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc220003520 con 0x7fc23806e5b0 2024-03-24T10:34:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc237fff640 1 -- 172.21.15.1:0/2961664551 >> v1:172.21.15.3:6789/0 conn(0x7fc2380b4570 legacy=0x7fc2380b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc237fff640 1 -- 172.21.15.1:0/2961664551 >> v1:172.21.15.1:6789/0 conn(0x7fc23806e5b0 legacy=0x7fc23806e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.859+0000 7fc237fff640 1 -- 172.21.15.1:0/2961664551 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2380b3d70 con 0x7fc23815c4d0 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc237fff640 1 --2- 172.21.15.1:0/2961664551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23815c4d0 0x7fc23815c960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc230009d90 tx=0x7fc23002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619e7084cb62002d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc2367fc640 1 -- 172.21.15.1:0/2961664551 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23002cd00 con 0x7fc23815c4d0 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc2367fc640 1 -- 172.21.15.1:0/2961664551 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23002ce60 con 0x7fc23815c4d0 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc2367fc640 1 -- 172.21.15.1:0/2961664551 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230034640 con 0x7fc23815c4d0 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 -- 172.21.15.1:0/2961664551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23815c4d0 msgr2=0x7fc23815c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/2961664551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23815c4d0 0x7fc23815c960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc230009d90 tx=0x7fc23002bf50 comp rx=0 tx=0).stop 2024-03-24T10:34:37.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 -- 172.21.15.1:0/2961664551 shutdown_connections 2024-03-24T10:34:37.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/2961664551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23815c4d0 0x7fc23815c960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 -- 172.21.15.1:0/2961664551 >> 172.21.15.1:0/2961664551 conn(0x7fc2380b1710 msgr2=0x7fc2380b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:37.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 -- 172.21.15.1:0/2961664551 shutdown_connections 2024-03-24T10:34:37.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.860+0000 7fc23ddd7640 1 -- 172.21.15.1:0/2961664551 wait complete. 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.861+0000 7fc23ddd7640 1 Processor -- start 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 -- start start 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc23806e5b0 0x7fc23815a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 0x7fc23813c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc238140950 con 0x7fc23815c4d0 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc23815a880 con 0x7fc2380b4570 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc23ddd7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc23813fd40 con 0x7fc23806e5b0 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.862+0000 7fc237fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43636/0 (socket says 172.21.15.1:43636) 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 -- 172.21.15.1:0/5854402 learned_addr learned my addr 172.21.15.1:0/5854402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc236ffd640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 0x7fc23813c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc2377fe640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc23806e5b0 0x7fc23815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 -- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc23806e5b0 msgr2=0x7fc23815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc23806e5b0 0x7fc23815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 -- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 msgr2=0x7fc23813c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 0x7fc23813c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 -- 172.21.15.1:0/5854402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc230009a70 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc236ffd640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 0x7fc23813c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc237fff640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc230009d90 tx=0x7fc230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230003b90 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc230003cf0 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230004250 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc238140020 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.863+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2381405e0 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.864+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23813dff0 con 0x7fc23815c4d0 2024-03-24T10:34:37.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.865+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc23003b040 con 0x7fc23815c4d0 2024-03-24T10:34:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.865+0000 7fc21ffff640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 0x7fc22406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.865+0000 7fc2377fe640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 0x7fc22406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.865+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fc2300ac310 con 0x7fc23815c4d0 2024-03-24T10:34:37.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.866+0000 7fc2377fe640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 0x7fc22406dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2280011e0 tx=0x7fc228007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:37.869+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc230076db0 con 0x7fc23815c4d0 2024-03-24T10:34:38.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.049+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc23804f4e0 con 0x7fc23815c4d0 2024-03-24T10:34:38.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.050+0000 7fc21ffff640 1 -- 172.21.15.1:0/5854402 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fc230009450 con 0x7fc23815c4d0 2024-03-24T10:34:38.050 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 msgr2=0x7fc22406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 0x7fc22406dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc2280011e0 tx=0x7fc228007d30 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 msgr2=0x7fc238143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc230009d90 tx=0x7fc230004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 shutdown_connections 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc22406b810 0x7fc22406dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc23815c4d0 0x7fc238143570 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2380b4570 0x7fc23813c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.051+0000 7fc23ddd7640 1 --2- 172.21.15.1:0/5854402 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc23806e5b0 0x7fc23815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.052+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 >> 172.21.15.1:0/5854402 conn(0x7fc2380b1710 msgr2=0x7fc23806d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.052+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 shutdown_connections 2024-03-24T10:34:38.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.052+0000 7fc23ddd7640 1 -- 172.21.15.1:0/5854402 wait complete. 2024-03-24T10:34:38.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:38.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.164+0000 7fe992a15640 1 Processor -- start 2024-03-24T10:34:38.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.164+0000 7fe992a15640 1 -- start start 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe98c146ee0 0x7fe98c071040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c06f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe98c06e390 0x7fe98c06e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe98c06fa50 con 0x7fe98c06e390 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe98c06ed40 con 0x7fe98c146ee0 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe992a15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe98c0b2f30 con 0x7fe98c06f0d0 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe990fb2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe98c06e390 0x7fe98c06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c06f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe990fb2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe98c06e390 0x7fe98c06e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43652/0 (socket says 172.21.15.1:43652) 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe98bfff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe98c146ee0 0x7fe98c071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.165+0000 7fe990fb2640 1 -- 172.21.15.1:0/1972276046 learned_addr learned my addr 172.21.15.1:0/1972276046 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 -- 172.21.15.1:0/1972276046 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 msgr2=0x7fe98c06f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:38.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c06f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 -- 172.21.15.1:0/1972276046 >> v2:172.21.15.3:3300/0 conn(0x7fe98c146ee0 msgr2=0x7fe98c071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 --2- 172.21.15.1:0/1972276046 >> v2:172.21.15.3:3300/0 conn(0x7fe98c146ee0 0x7fe98c071040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 -- 172.21.15.1:0/1972276046 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c0b3180 con 0x7fe98c06e390 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe98b7fe640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c06f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.166+0000 7fe990fb2640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c06e800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe98c06cbc0 tx=0x7fe98402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bcba56dde96b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe98affd640 1 -- 172.21.15.1:0/1972276046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98402c9e0 con 0x7fe98c06e390 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe98affd640 1 -- 172.21.15.1:0/1972276046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98402cb40 con 0x7fe98c06e390 2024-03-24T10:34:38.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe98affd640 1 -- 172.21.15.1:0/1972276046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840343f0 con 0x7fe98c06e390 2024-03-24T10:34:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 -- 172.21.15.1:0/1972276046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 msgr2=0x7fe98c06e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c06e800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe98c06cbc0 tx=0x7fe98402be40 comp rx=0 tx=0).stop 2024-03-24T10:34:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 -- 172.21.15.1:0/1972276046 shutdown_connections 2024-03-24T10:34:38.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c06e800 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 --2- 172.21.15.1:0/1972276046 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c06f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 --2- 172.21.15.1:0/1972276046 >> v2:172.21.15.3:3300/0 conn(0x7fe98c146ee0 0x7fe98c071040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 -- 172.21.15.1:0/1972276046 >> 172.21.15.1:0/1972276046 conn(0x7fe98c0cff00 msgr2=0x7fe98c0d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 -- 172.21.15.1:0/1972276046 shutdown_connections 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.167+0000 7fe992a15640 1 -- 172.21.15.1:0/1972276046 wait complete. 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.168+0000 7fe992a15640 1 Processor -- start 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 -- start start 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c13e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe98c146ee0 0x7fe98c071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1f7530 con 0x7fe98c06e390 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c13ed30 con 0x7fe98c146ee0 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe992a15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe98c1f77d0 con 0x7fe98c06f0d0 2024-03-24T10:34:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43668/0 (socket says 172.21.15.1:43668) 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe98bfff640 1 -- 172.21.15.1:0/1811004471 learned_addr learned my addr 172.21.15.1:0/1811004471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe990fb2640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe98c146ee0 0x7fe98c071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.169+0000 7fe98b7fe640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c13e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 -- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 msgr2=0x7fe98c13e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c13e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 -- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe98c146ee0 msgr2=0x7fe98c071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe98c146ee0 0x7fe98c071ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 -- 172.21.15.1:0/1811004471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe980009400 con 0x7fe98c06e390 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe98bfff640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00e870 tx=0x7fe97c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0094f0 con 0x7fe98c06e390 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c00c940 con 0x7fe98c06e390 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe984009b30 con 0x7fe98c06e390 2024-03-24T10:34:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00ccf0 con 0x7fe98c06e390 2024-03-24T10:34:38.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.170+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98c13ea20 con 0x7fe98c06e390 2024-03-24T10:34:38.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.171+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98c04f4e0 con 0x7fe98c06e390 2024-03-24T10:34:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.172+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe97c013470 con 0x7fe98c06e390 2024-03-24T10:34:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.172+0000 7fe988ff9640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 0x7fe97806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.172+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fe97c08a2b0 con 0x7fe98c06e390 2024-03-24T10:34:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.172+0000 7fe98b7fe640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 0x7fe97806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.172+0000 7fe98b7fe640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 0x7fe97806dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe98c1e9cc0 tx=0x7fe980009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.176+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe97c054cc0 con 0x7fe98c06e390 2024-03-24T10:34:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.371+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe98c0afb60 con 0x7fe98c06e390 2024-03-24T10:34:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.371+0000 7fe988ff9640 1 -- 172.21.15.1:0/1811004471 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fe97c058380 con 0x7fe98c06e390 2024-03-24T10:34:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 msgr2=0x7fe97806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 0x7fe97806dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe98c1e9cc0 tx=0x7fe980009190 comp rx=0 tx=0).stop 2024-03-24T10:34:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 msgr2=0x7fe98c1f7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00e870 tx=0x7fe97c00ed30 comp rx=0 tx=0).stop 2024-03-24T10:34:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 shutdown_connections 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe97806b810 0x7fe97806dcd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe98c146ee0 0x7fe98c071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe98c06f0d0 0x7fe98c13e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 --2- 172.21.15.1:0/1811004471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe98c06e390 0x7fe98c1f7fd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 >> 172.21.15.1:0/1811004471 conn(0x7fe98c0cff00 msgr2=0x7fe98c0d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 shutdown_connections 2024-03-24T10:34:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.373+0000 7fe992a15640 1 -- 172.21.15.1:0/1811004471 wait complete. 2024-03-24T10:34:38.384 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:34:38.384 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:34:38.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:38.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 Processor -- start 2024-03-24T10:34:38.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 -- start start 2024-03-24T10:34:38.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f061c0b74c0 0x7f061c0d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 0x7f061c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c0b4c70 con 0x7f061c06c770 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.502+0000 7f06242df640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f061c0d1340 con 0x7f061c0b74c0 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f06242df640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f061c0b4f50 con 0x7f061c06d2d0 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f061c0b74c0 0x7f061c0d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062287c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 0x7f061c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f061c0b74c0 0x7f061c0d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:49080/0 (socket says 172.21.15.1:49080) 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062187a640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f061c06c770 0x7f061c06cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:34128/0 (socket says 172.21.15.1:34128) 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 -- 172.21.15.1:0/117824036 learned_addr learned my addr 172.21.15.1:0/117824036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 -- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 msgr2=0x7f061c15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 0x7f061c15d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 -- 172.21.15.1:0/117824036 >> v1:172.21.15.3:6789/0 conn(0x7f061c06c770 legacy=0x7f061c06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062207b640 1 -- 172.21.15.1:0/117824036 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0b5230 con 0x7f061c0b74c0 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f0621079640 1 -- 172.21.15.1:0/117824036 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264223742 0 0) 0x7f061c0b4c70 con 0x7f061c06c770 2024-03-24T10:34:38.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.503+0000 7f062287c640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 0x7f061c15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f062207b640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0b74c0 0x7f061c0d09d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0618009870 tx=0x7f061802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2342d46de406179e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f0621079640 1 -- 172.21.15.1:0/117824036 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618030840 con 0x7f061c0b74c0 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f0621079640 1 -- 172.21.15.1:0/117824036 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06180309a0 con 0x7f061c0b74c0 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f0621079640 1 -- 172.21.15.1:0/117824036 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618030d30 con 0x7f061c0b74c0 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f06242df640 1 -- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0b74c0 msgr2=0x7f061c0d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.504+0000 7f06242df640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0b74c0 0x7f061c0d09d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0618009870 tx=0x7f061802b980 comp rx=0 tx=0).stop 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 -- 172.21.15.1:0/117824036 shutdown_connections 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c06d2d0 0x7f061c15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 --2- 172.21.15.1:0/117824036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c0b74c0 0x7f061c0d09d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 -- 172.21.15.1:0/117824036 >> 172.21.15.1:0/117824036 conn(0x7f061c0d1c80 msgr2=0x7f061c0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 -- 172.21.15.1:0/117824036 shutdown_connections 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.505+0000 7f06242df640 1 -- 172.21.15.1:0/117824036 wait complete. 2024-03-24T10:34:38.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 Processor -- start 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 -- start start 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 0x7f061c152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 0x7f061c09e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c09df40 con 0x7f061c06c770 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f061c09cc80 con 0x7f061c06d2d0 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.506+0000 7f06242df640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f061c09cf20 con 0x7f061c0b74c0 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43680/0 (socket says 172.21.15.1:43680) 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062287c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 0x7f061c09e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 -- 172.21.15.1:0/751660856 learned_addr learned my addr 172.21.15.1:0/751660856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062187a640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 0x7f061c152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 -- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 msgr2=0x7f061c09e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 0x7f061c09e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 -- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 msgr2=0x7f061c152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 0x7f061c152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 -- 172.21.15.1:0/751660856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0614009a70 con 0x7f061c06c770 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062287c640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 0x7f061c09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062187a640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 0x7f061c152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f062207b640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0618009870 tx=0x7f0618004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061803e040 con 0x7f061c06c770 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0618039c90 con 0x7f061c06c770 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06180094d0 con 0x7f061c06c770 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618004030 con 0x7f061c06c770 2024-03-24T10:34:38.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.507+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c09d4e0 con 0x7f061c06c770 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.508+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c04f4e0 con 0x7f061c06c770 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.509+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06180041b0 con 0x7f061c06c770 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.509+0000 7f0612ffd640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 0x7f060006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.509+0000 7f062187a640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 0x7f060006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.509+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f06180ac790 con 0x7f061c06c770 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.510+0000 7f062187a640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 0x7f060006dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f061c15efc0 tx=0x7f060c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.513+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0618077350 con 0x7f061c06c770 2024-03-24T10:34:38.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.695+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f061c0b7c00 con 0x7f061c06c770 2024-03-24T10:34:38.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.696+0000 7f0612ffd640 1 -- 172.21.15.1:0/751660856 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16832 (secure 0 0 0) 0x7f061807aa10 con 0x7f061c06c770 2024-03-24T10:34:38.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:38.697 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:36.249848+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 msgr2=0x7f060006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 0x7f060006dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f061c15efc0 tx=0x7f060c007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 msgr2=0x7f061c09da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0618009870 tx=0x7f0618004730 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 shutdown_connections 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f060006b840 0x7f060006dd00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f061c0b74c0 0x7f061c09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f061c06d2d0 0x7f061c152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 --2- 172.21.15.1:0/751660856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c06c770 0x7f061c09da00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 >> 172.21.15.1:0/751660856 conn(0x7f061c0d1c80 msgr2=0x7f061c06fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 shutdown_connections 2024-03-24T10:34:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.698+0000 7f06242df640 1 -- 172.21.15.1:0/751660856 wait complete. 2024-03-24T10:34:38.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:38.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:38.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.813+0000 7f32a3b93640 1 Processor -- start 2024-03-24T10:34:38.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.813+0000 7f32a3b93640 1 -- start start 2024-03-24T10:34:38.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c06b310 0x7f329c15ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f329c06b6f0 0x7f329c0b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 0x7f329c0cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f329c0b6030 con 0x7f329c06b310 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f329c0cfa00 con 0x7f329c06b6f0 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a3b93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f329c15f960 con 0x7f329c0cf010 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a192f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c06b310 0x7f329c15ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a192f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c06b310 0x7f329c15ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43690/0 (socket says 172.21.15.1:43690) 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a192f640 1 -- 172.21.15.1:0/1985337958 learned_addr learned my addr 172.21.15.1:0/1985337958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a2130640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 0x7f329c0cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.814+0000 7f32a112e640 1 --2- 172.21.15.1:0/1985337958 >> v2:172.21.15.3:3300/0 conn(0x7f329c06b6f0 0x7f329c0b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 -- 172.21.15.1:0/1985337958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 msgr2=0x7f329c0cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 0x7f329c0cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 -- 172.21.15.1:0/1985337958 >> v2:172.21.15.3:3300/0 conn(0x7f329c06b6f0 msgr2=0x7f329c0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 --2- 172.21.15.1:0/1985337958 >> v2:172.21.15.3:3300/0 conn(0x7f329c06b6f0 0x7f329c0b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 -- 172.21.15.1:0/1985337958 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c15fb70 con 0x7f329c06b310 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a2130640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 0x7f329c0cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.815+0000 7f32a192f640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c06b310 0x7f329c15ce80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f329c0d1c00 tx=0x7f329402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a8cfc52721361b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a092d640 1 -- 172.21.15.1:0/1985337958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329402ca70 con 0x7f329c06b310 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a092d640 1 -- 172.21.15.1:0/1985337958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329402cbd0 con 0x7f329c06b310 2024-03-24T10:34:38.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a092d640 1 -- 172.21.15.1:0/1985337958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32940343f0 con 0x7f329c06b310 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 -- 172.21.15.1:0/1985337958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c06b310 msgr2=0x7f329c15ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c06b310 0x7f329c15ce80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f329c0d1c00 tx=0x7f329402bed0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 -- 172.21.15.1:0/1985337958 shutdown_connections 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c0cf010 0x7f329c0cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1985337958 >> v2:172.21.15.3:3300/0 conn(0x7f329c06b6f0 0x7f329c0b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1985337958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c06b310 0x7f329c15ce80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 -- 172.21.15.1:0/1985337958 >> 172.21.15.1:0/1985337958 conn(0x7f329c06c1b0 msgr2=0x7f329c06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 -- 172.21.15.1:0/1985337958 shutdown_connections 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.816+0000 7f32a3b93640 1 -- 172.21.15.1:0/1985337958 wait complete. 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.817+0000 7f32a3b93640 1 Processor -- start 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 -- start start 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c06b310 0x7f329c159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f329c06b6f0 0x7f329c147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f329c15aa60 con 0x7f329c0cf010 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f329c159e50 con 0x7f329c06b6f0 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a3b93640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f329c15a130 con 0x7f329c06b310 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a2130640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a192f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c06b310 0x7f329c159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a2130640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43704/0 (socket says 172.21.15.1:43704) 2024-03-24T10:34:38.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a2130640 1 -- 172.21.15.1:0/1856641613 learned_addr learned my addr 172.21.15.1:0/1856641613 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.818+0000 7f32a112e640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f329c06b6f0 0x7f329c147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 -- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c06b310 msgr2=0x7f329c159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c06b310 0x7f329c159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 -- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f329c06b6f0 msgr2=0x7f329c147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f329c06b6f0 0x7f329c147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 -- 172.21.15.1:0/1856641613 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c009400 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a2130640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e880 tx=0x7f328c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c009650 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328c004020 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3294009b30 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c0043b0 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.819+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c15a6f0 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.820+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c0b5960 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.820+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f328c00c940 con 0x7f329c0cf010 2024-03-24T10:34:38.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.821+0000 7f329a7fc640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 0x7f328806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.821+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f328c089ff0 con 0x7f329c0cf010 2024-03-24T10:34:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.821+0000 7f32a192f640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 0x7f328806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.821+0000 7f32a192f640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 0x7f328806dcd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3294035000 tx=0x7f3294030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:38.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:38.825+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f328c0549c0 con 0x7f329c0cf010 2024-03-24T10:34:39.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.004+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f329c04f4e0 con 0x7f329c0cf010 2024-03-24T10:34:39.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.005+0000 7f329a7fc640 1 -- 172.21.15.1:0/1856641613 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f328c058080 con 0x7f329c0cf010 2024-03-24T10:34:39.005 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 msgr2=0x7f328806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 0x7f328806dcd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3294035000 tx=0x7f3294030000 comp rx=0 tx=0).stop 2024-03-24T10:34:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 msgr2=0x7f329c154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e880 tx=0x7f328c00ed40 comp rx=0 tx=0).stop 2024-03-24T10:34:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 shutdown_connections 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f328806b810 0x7f328806dcd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cf010 0x7f329c154600 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f329c06b6f0 0x7f329c147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 --2- 172.21.15.1:0/1856641613 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f329c06b310 0x7f329c159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 >> 172.21.15.1:0/1856641613 conn(0x7f329c06c1b0 msgr2=0x7f329c0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 shutdown_connections 2024-03-24T10:34:39.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.007+0000 7f32a3b93640 1 -- 172.21.15.1:0/1856641613 wait complete. 2024-03-24T10:34:39.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:39.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.121+0000 7ffb27d81640 1 Processor -- start 2024-03-24T10:34:39.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.122+0000 7ffb27d81640 1 -- start start 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.122+0000 7ffb27d81640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb201615f0 0x7ffb200d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.122+0000 7ffb27d81640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb200c7a40 con 0x7ffb200b4e40 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.122+0000 7ffb27d81640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb200b55b0 con 0x7ffb20162610 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.122+0000 7ffb27d81640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7ffb200cfd50 con 0x7ffb201615f0 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb2531c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb200b4e40 0x7ffb20161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52882/0 (socket says 172.21.15.1:52882) 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb2531c640 1 -- 172.21.15.1:0/3482706637 learned_addr learned my addr 172.21.15.1:0/3482706637 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb25b1d640 1 --2- 172.21.15.1:0/3482706637 >> v2:172.21.15.3:3300/0 conn(0x7ffb201615f0 0x7ffb200d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb24b1b640 1 -- 172.21.15.1:0/3482706637 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197867619 0 0) 0x7ffb200c7a40 con 0x7ffb200b4e40 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb25b1d640 1 -- 172.21.15.1:0/3482706637 >> v1:172.21.15.3:6789/0 conn(0x7ffb20162610 legacy=0x7ffb200d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb25b1d640 1 -- 172.21.15.1:0/3482706637 >> v1:172.21.15.1:6789/0 conn(0x7ffb200b4e40 legacy=0x7ffb20161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.123+0000 7ffb25b1d640 1 -- 172.21.15.1:0/3482706637 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb200b52d0 con 0x7ffb201615f0 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb25b1d640 1 --2- 172.21.15.1:0/3482706637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb201615f0 0x7ffb200d4a10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009720 tx=0x7ffb1802b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c79f0b9019a68d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb24b1b640 1 -- 172.21.15.1:0/3482706637 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18030710 con 0x7ffb201615f0 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb24b1b640 1 -- 172.21.15.1:0/3482706637 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb1802e050 con 0x7ffb201615f0 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb24b1b640 1 -- 172.21.15.1:0/3482706637 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18030a90 con 0x7ffb201615f0 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb27d81640 1 -- 172.21.15.1:0/3482706637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb201615f0 msgr2=0x7ffb200d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb27d81640 1 --2- 172.21.15.1:0/3482706637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb201615f0 0x7ffb200d4a10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009720 tx=0x7ffb1802b730 comp rx=0 tx=0).stop 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb27d81640 1 -- 172.21.15.1:0/3482706637 shutdown_connections 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb27d81640 1 --2- 172.21.15.1:0/3482706637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb201615f0 0x7ffb200d4a10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.124+0000 7ffb27d81640 1 -- 172.21.15.1:0/3482706637 >> 172.21.15.1:0/3482706637 conn(0x7ffb200d3ef0 msgr2=0x7ffb200d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.125+0000 7ffb27d81640 1 -- 172.21.15.1:0/3482706637 shutdown_connections 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.125+0000 7ffb27d81640 1 -- 172.21.15.1:0/3482706637 wait complete. 2024-03-24T10:34:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 Processor -- start 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 -- start start 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 0x7ffb200cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 0x7ffb2014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2014e7e0 con 0x7ffb20158470 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ffb200b8190 con 0x7ffb20162610 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb27d81640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb200b8400 con 0x7ffb201615f0 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb2631e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb25b1d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 0x7ffb200cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.126+0000 7ffb2631e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43708/0 (socket says 172.21.15.1:43708) 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 -- 172.21.15.1:0/824298880 learned_addr learned my addr 172.21.15.1:0/824298880 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2531c640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 0x7ffb2014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 -- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 msgr2=0x7ffb200cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 0x7ffb200cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 -- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 msgr2=0x7ffb2014e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 0x7ffb2014e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 -- 172.21.15.1:0/824298880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb18009400 con 0x7ffb20158470 2024-03-24T10:34:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2531c640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 0x7ffb2014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb25b1d640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 0x7ffb200cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb2631e640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000c8a0 tx=0x7ffb1000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003da0 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb10005070 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb200b86e0 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10005400 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.127+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb200ce1f0 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.128+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb200b8c00 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.129+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb10005560 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.129+0000 7ffb1e7fc640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 0x7ffb0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.129+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7ffb10089af0 con 0x7ffb20158470 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.129+0000 7ffb25b1d640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 0x7ffb0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.129+0000 7ffb25b1d640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 0x7ffb0c06dcd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffb18034000 tx=0x7ffb18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.133+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb200b8c00 con 0x7ffb20158470 2024-03-24T10:34:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.315+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb200b8c00 con 0x7ffb20158470 2024-03-24T10:34:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.315+0000 7ffb1e7fc640 1 -- 172.21.15.1:0/824298880 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7ffb200b8c00 con 0x7ffb20158470 2024-03-24T10:34:39.316 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 msgr2=0x7ffb0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 0x7ffb0c06dcd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffb18034000 tx=0x7ffb18008030 comp rx=0 tx=0).stop 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 msgr2=0x7ffb201f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000c8a0 tx=0x7ffb1000cd60 comp rx=0 tx=0).stop 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 shutdown_connections 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb0c06b810 0x7ffb0c06dcd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb20158470 0x7ffb201f4200 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb20162610 0x7ffb2014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 --2- 172.21.15.1:0/824298880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb201615f0 0x7ffb200cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.317+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 >> 172.21.15.1:0/824298880 conn(0x7ffb200d3ef0 msgr2=0x7ffb200d29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.318+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 shutdown_connections 2024-03-24T10:34:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.318+0000 7ffb27d81640 1 -- 172.21.15.1:0/824298880 wait complete. 2024-03-24T10:34:39.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:39.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.432+0000 7f7fd8f3e640 1 Processor -- start 2024-03-24T10:34:39.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.432+0000 7f7fd8f3e640 1 -- start start 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd8f3e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7fd415e390 0x7f7fd40d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd8f3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd40b0b70 con 0x7f7fd40b2ed0 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd8f3e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fd40d1d60 con 0x7f7fd40cfae0 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd8f3e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f7fd40b0de0 con 0x7f7fd415e390 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fd40b2ed0 0x7f7fd40b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52892/0 (socket says 172.21.15.1:52892) 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd1d9b640 1 -- 172.21.15.1:0/815150472 learned_addr learned my addr 172.21.15.1:0/815150472 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.433+0000 7f7fd259c640 1 --2- 172.21.15.1:0/815150472 >> v2:172.21.15.3:3300/0 conn(0x7f7fd415e390 0x7f7fd40d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549375089 0 0) 0x7f7fd40b0b70 con 0x7f7fd40b2ed0 2024-03-24T10:34:39.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fc0003520 con 0x7f7fd40b2ed0 2024-03-24T10:34:39.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd259c640 1 -- 172.21.15.1:0/815150472 >> v1:172.21.15.3:6789/0 conn(0x7f7fd40cfae0 legacy=0x7f7fd40b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407973924 0 0) 0x7f7fd40d1d60 con 0x7f7fd40cfae0 2024-03-24T10:34:39.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd259c640 1 -- 172.21.15.1:0/815150472 >> v1:172.21.15.1:6789/0 conn(0x7f7fd40b2ed0 legacy=0x7f7fd40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.434+0000 7f7fd259c640 1 -- 172.21.15.1:0/815150472 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd415e770 con 0x7f7fd415e390 2024-03-24T10:34:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd259c640 1 --2- 172.21.15.1:0/815150472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd415e390 0x7f7fd40d2310 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0097a0 tx=0x7f7fcc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6e7334aae180414 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc030610 con 0x7f7fd415e390 2024-03-24T10:34:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc030770 con 0x7f7fd415e390 2024-03-24T10:34:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd159a640 1 -- 172.21.15.1:0/815150472 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc030b00 con 0x7f7fd415e390 2024-03-24T10:34:39.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/815150472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd415e390 msgr2=0x7f7fd40d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/815150472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd415e390 0x7f7fd40d2310 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0097a0 tx=0x7f7fcc02b960 comp rx=0 tx=0).stop 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/815150472 shutdown_connections 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/815150472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd415e390 0x7f7fd40d2310 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/815150472 >> 172.21.15.1:0/815150472 conn(0x7f7fd40d3ee0 msgr2=0x7f7fd40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/815150472 shutdown_connections 2024-03-24T10:34:39.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.435+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/815150472 wait complete. 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 Processor -- start 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 -- start start 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 0x7f7fd415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd40cfae0 0x7f7fd40b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd415b930 con 0x7f7fd415e390 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd40b8450 con 0x7f7fd40cfae0 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.437+0000 7f7fd8f3e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd40b8730 con 0x7f7fd40b2ed0 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd40cfae0 0x7f7fd40b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43714/0 (socket says 172.21.15.1:43714) 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd259c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 0x7f7fd415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 -- 172.21.15.1:0/1024221272 learned_addr learned my addr 172.21.15.1:0/1024221272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 -- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 msgr2=0x7f7fd415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 0x7f7fd415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 -- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd40cfae0 msgr2=0x7f7fd40b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd40cfae0 0x7f7fd40b9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 -- 172.21.15.1:0/1024221272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fcc009400 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd259c640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 0x7f7fd415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd2d9d640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c760 tx=0x7f7fc400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4003da0 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc400ce70 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd40b8a10 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4005290 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.438+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd40b41a0 con 0x7f7fd415e390 2024-03-24T10:34:39.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.439+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd40b3d20 con 0x7f7fd415e390 2024-03-24T10:34:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.440+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7fc40053f0 con 0x7f7fd415e390 2024-03-24T10:34:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.441+0000 7f7fcaffd640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 0x7f7fb406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.441+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f7fc4089fb0 con 0x7f7fd415e390 2024-03-24T10:34:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.441+0000 7f7fd259c640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 0x7f7fb406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.441+0000 7f7fd259c640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 0x7f7fb406dd00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc036000 tx=0x7f7fcc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.445+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7fc40bde10 con 0x7f7fd415e390 2024-03-24T10:34:39.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.624+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7fd40b43c0 con 0x7f7fd415e390 2024-03-24T10:34:39.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.624+0000 7f7fcaffd640 1 -- 172.21.15.1:0/1024221272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f7fd40b3d20 con 0x7f7fd415e390 2024-03-24T10:34:39.625 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 msgr2=0x7f7fb406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 0x7f7fb406dd00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc036000 tx=0x7f7fcc008030 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 msgr2=0x7f7fd41f3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c760 tx=0x7f7fc400cc20 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 shutdown_connections 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7fb406b840 0x7f7fb406dd00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd415e390 0x7f7fd41f3cb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd40cfae0 0x7f7fd40b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 --2- 172.21.15.1:0/1024221272 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7fd40b2ed0 0x7f7fd415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 >> 172.21.15.1:0/1024221272 conn(0x7f7fd40d3ee0 msgr2=0x7f7fd40b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 shutdown_connections 2024-03-24T10:34:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.626+0000 7f7fd8f3e640 1 -- 172.21.15.1:0/1024221272 wait complete. 2024-03-24T10:34:39.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:34:39.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.739+0000 7fa581125640 1 Processor -- start 2024-03-24T10:34:39.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.739+0000 7fa581125640 1 -- start start 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa581125640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa57c0b74c0 0x7fa57c0d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa581125640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa57c0b4c70 con 0x7fa57c06c770 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa581125640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa57c0b4f50 con 0x7fa57c06d2d0 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa581125640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa57c0d1340 con 0x7fa57c0b74c0 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa57a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa57c06c770 0x7fa57c06cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52908/0 (socket says 172.21.15.1:52908) 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa57a59c640 1 -- 172.21.15.1:0/1996461895 learned_addr learned my addr 172.21.15.1:0/1996461895 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.740+0000 7fa57ad9d640 1 --2- 172.21.15.1:0/1996461895 >> v2:172.21.15.3:3300/0 conn(0x7fa57c0b74c0 0x7fa57c0d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa579d9b640 1 -- 172.21.15.1:0/1996461895 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333446563 0 0) 0x7fa57c0b4c70 con 0x7fa57c06c770 2024-03-24T10:34:39.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa579d9b640 1 -- 172.21.15.1:0/1996461895 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa568003520 con 0x7fa57c06c770 2024-03-24T10:34:39.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa57ad9d640 1 -- 172.21.15.1:0/1996461895 >> v1:172.21.15.3:6789/0 conn(0x7fa57c06d2d0 legacy=0x7fa57c15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa57ad9d640 1 -- 172.21.15.1:0/1996461895 >> v1:172.21.15.1:6789/0 conn(0x7fa57c06c770 legacy=0x7fa57c06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa57ad9d640 1 -- 172.21.15.1:0/1996461895 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c0b5230 con 0x7fa57c0b74c0 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.741+0000 7fa57ad9d640 1 --2- 172.21.15.1:0/1996461895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c0d09d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5740097a0 tx=0x7fa57402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9515e72e4da40369 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa579d9b640 1 -- 172.21.15.1:0/1996461895 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574030610 con 0x7fa57c0b74c0 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa579d9b640 1 -- 172.21.15.1:0/1996461895 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa574030770 con 0x7fa57c0b74c0 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa579d9b640 1 -- 172.21.15.1:0/1996461895 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574030b00 con 0x7fa57c0b74c0 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 -- 172.21.15.1:0/1996461895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 msgr2=0x7fa57c0d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 --2- 172.21.15.1:0/1996461895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c0d09d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5740097a0 tx=0x7fa57402b960 comp rx=0 tx=0).stop 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 -- 172.21.15.1:0/1996461895 shutdown_connections 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 --2- 172.21.15.1:0/1996461895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c0d09d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 -- 172.21.15.1:0/1996461895 >> 172.21.15.1:0/1996461895 conn(0x7fa57c0d1c80 msgr2=0x7fa57c0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 -- 172.21.15.1:0/1996461895 shutdown_connections 2024-03-24T10:34:39.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.742+0000 7fa581125640 1 -- 172.21.15.1:0/1996461895 wait complete. 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 Processor -- start 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 -- start start 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 0x7fa57c1f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c09f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c155720 con 0x7fa57c06d2d0 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c0b2840 con 0x7fa57c0b74c0 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa581125640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa57c0b1c30 con 0x7fa57c06c770 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa57a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.744+0000 7fa57ad9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 0x7fa57c1f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43720/0 (socket says 172.21.15.1:43720) 2024-03-24T10:34:39.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 -- 172.21.15.1:0/187908619 learned_addr learned my addr 172.21.15.1:0/187908619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57b59e640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c09f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 -- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 msgr2=0x7fa57c1f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 0x7fa57c1f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 -- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 msgr2=0x7fa57c09f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c09f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 -- 172.21.15.1:0/187908619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa574009400 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57ad9d640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 0x7fa57c1f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa57a59c640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c500 tx=0x7fa56400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564005ce0 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa564005e40 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c0b1f10 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564007330 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.745+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c0b24d0 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.746+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57c09bbd0 con 0x7fa57c06d2d0 2024-03-24T10:34:39.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.747+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa564007550 con 0x7fa57c06d2d0 2024-03-24T10:34:39.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.747+0000 7fa5737fe640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 0x7fa55c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:39.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.747+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fa564089840 con 0x7fa57c06d2d0 2024-03-24T10:34:39.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.747+0000 7fa57ad9d640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 0x7fa55c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:39.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.748+0000 7fa57ad9d640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 0x7fa55c06dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:39.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.752+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa564054320 con 0x7fa57c06d2d0 2024-03-24T10:34:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.946+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa57c04f4e0 con 0x7fa57c06d2d0 2024-03-24T10:34:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.946+0000 7fa5737fe640 1 -- 172.21.15.1:0/187908619 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fa564017080 con 0x7fa57c06d2d0 2024-03-24T10:34:39.947 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 msgr2=0x7fa55c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 0x7fa55c06dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403b040 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 msgr2=0x7fa57c1551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c500 tx=0x7fa56400c9c0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 shutdown_connections 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa55c06b810 0x7fa55c06dcd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57c0b74c0 0x7fa57c09f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa57c06d2d0 0x7fa57c1551e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 --2- 172.21.15.1:0/187908619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa57c06c770 0x7fa57c1f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 >> 172.21.15.1:0/187908619 conn(0x7fa57c0d1c80 msgr2=0x7fa57c161090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 shutdown_connections 2024-03-24T10:34:39.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:39.948+0000 7fa581125640 1 -- 172.21.15.1:0/187908619 wait complete. 2024-03-24T10:34:39.959 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi003.front.sepia.ceph.com... 2024-03-24T10:34:39.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:39.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-24T10:34:39.987 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:39.987 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:34:39.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:34:39.987 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T10:34:40.042 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T10:34:40.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.144+0000 7feed3fa1640 1 Processor -- start 2024-03-24T10:34:40.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.144+0000 7feed3fa1640 1 -- start start 2024-03-24T10:34:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed3fa1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feecc15e390 0x7feecc0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed3fa1640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feecc0b4ba0 0x7feecc0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed3fa1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feecc15e770 con 0x7feecc0d1000 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed3fa1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7feecc0b3190 con 0x7feecc15e390 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed3fa1640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7feecc0d4e40 con 0x7feecc0b4ba0 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed1d3d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feecc15e390 0x7feecc0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed153c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feecc0d1000 0x7feecc0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52920/0 (socket says 172.21.15.1:52920) 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed153c640 1 -- 172.21.15.1:0/1079855483 learned_addr learned my addr 172.21.15.1:0/1079855483 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed253e640 1 --2- 172.21.15.1:0/1079855483 >> v2:172.21.15.3:3300/0 conn(0x7feecc0b4ba0 0x7feecc0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed1d3d640 1 -- 172.21.15.1:0/1079855483 >> v2:172.21.15.3:3300/0 conn(0x7feecc0b4ba0 msgr2=0x7feecc0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed1d3d640 1 --2- 172.21.15.1:0/1079855483 >> v2:172.21.15.3:3300/0 conn(0x7feecc0b4ba0 0x7feecc0d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed1d3d640 1 -- 172.21.15.1:0/1079855483 >> v1:172.21.15.1:6789/0 conn(0x7feecc0d1000 legacy=0x7feecc0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed1d3d640 1 -- 172.21.15.1:0/1079855483 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc0d1e40 con 0x7feecc15e390 2024-03-24T10:34:40.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.145+0000 7feed0d3b640 1 -- 172.21.15.1:0/1079855483 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710497900 0 0) 0x7feebc001110 con 0x7feecc0d1000 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.146+0000 7feed1d3d640 1 --2- 172.21.15.1:0/1079855483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc15e390 0x7feecc0d24a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feecc08e5b0 tx=0x7feec802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ddbb4454ec95be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.146+0000 7feed0d3b640 1 -- 172.21.15.1:0/1079855483 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8031710 con 0x7feecc15e390 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.146+0000 7feed0d3b640 1 -- 172.21.15.1:0/1079855483 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec802f050 con 0x7feecc15e390 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.146+0000 7feed0d3b640 1 -- 172.21.15.1:0/1079855483 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8031a90 con 0x7feecc15e390 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 -- 172.21.15.1:0/1079855483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc15e390 msgr2=0x7feecc0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 --2- 172.21.15.1:0/1079855483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc15e390 0x7feecc0d24a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feecc08e5b0 tx=0x7feec802b7b0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 -- 172.21.15.1:0/1079855483 shutdown_connections 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 --2- 172.21.15.1:0/1079855483 >> v2:172.21.15.3:3300/0 conn(0x7feecc0b4ba0 0x7feecc0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 --2- 172.21.15.1:0/1079855483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc15e390 0x7feecc0d24a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 -- 172.21.15.1:0/1079855483 >> 172.21.15.1:0/1079855483 conn(0x7feecc0d3ec0 msgr2=0x7feecc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 -- 172.21.15.1:0/1079855483 shutdown_connections 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.147+0000 7feed3fa1640 1 -- 172.21.15.1:0/1079855483 wait complete. 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.148+0000 7feed3fa1640 1 Processor -- start 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.148+0000 7feed3fa1640 1 -- start start 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feecc0d1000 0x7feecc1f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 0x7feecc14daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feecc1f6e50 con 0x7feecc0b4ba0 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7feecc14e030 con 0x7feecc0d1000 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7feecc15aa70 con 0x7feecc15e390 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43726/0 (socket says 172.21.15.1:43726) 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 -- 172.21.15.1:0/3484627865 learned_addr learned my addr 172.21.15.1:0/3484627865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:40.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed153c640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feecc0d1000 0x7feecc1f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed253e640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 0x7feecc14daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 -- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 msgr2=0x7feecc14daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 0x7feecc14daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 -- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feecc0d1000 msgr2=0x7feecc1f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feecc0d1000 0x7feecc1f6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 -- 172.21.15.1:0/3484627865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec8009400 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed253e640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 0x7feecc14daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed1d3d640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feec8000c00 tx=0x7feec80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec803c810 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec803c970 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feecc15ad50 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec803cd20 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.149+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feecc0cceb0 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.150+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feecc15aee0 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.151+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feec803a040 con 0x7feecc0b4ba0 2024-03-24T10:34:40.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.151+0000 7feec27fc640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 0x7feeb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.151+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7feec80aba40 con 0x7feecc0b4ba0 2024-03-24T10:34:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.151+0000 7feed153c640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 0x7feeb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.152+0000 7feed153c640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 0x7feeb006dcd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feecc06d300 tx=0x7feebc007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.155+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feec80ade60 con 0x7feecc0b4ba0 2024-03-24T10:34:40.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.374+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feecc0d2280 con 0x7feecc0b4ba0 2024-03-24T10:34:40.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.375+0000 7feec27fc640 1 -- 172.21.15.1:0/3484627865 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7feec8080070 con 0x7feecc0b4ba0 2024-03-24T10:34:40.375 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:40.375 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:40.376 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:40.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 msgr2=0x7feeb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 0x7feeb006dcd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feecc06d300 tx=0x7feebc007b90 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 msgr2=0x7feecc0cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feec8000c00 tx=0x7feec80046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 shutdown_connections 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feeb006b810 0x7feeb006dcd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feecc15e390 0x7feecc14daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feecc0d1000 0x7feecc1f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 --2- 172.21.15.1:0/3484627865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feecc0b4ba0 0x7feecc0cb0b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 >> 172.21.15.1:0/3484627865 conn(0x7feecc0d3ec0 msgr2=0x7feecc0d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 shutdown_connections 2024-03-24T10:34:40.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.377+0000 7feed3fa1640 1 -- 172.21.15.1:0/3484627865 wait complete. 2024-03-24T10:34:40.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.489+0000 7f60c3455640 1 Processor -- start 2024-03-24T10:34:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.489+0000 7f60c3455640 1 -- start start 2024-03-24T10:34:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c3455640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f60bc0d00d0 0x7f60bc166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c3455640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60bc06ce40 con 0x7f60bc15bda0 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c3455640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60bc06d120 con 0x7f60bc06c910 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c3455640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f60bc0b2db0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f60bc0d00d0 0x7f60bc166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f60bc0d00d0 0x7f60bc166a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43740/0 (socket says 172.21.15.1:43740) 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c11f1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60bc15bda0 0x7f60bc15ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52932/0 (socket says 172.21.15.1:52932) 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 -- 172.21.15.1:0/1799684592 learned_addr learned my addr 172.21.15.1:0/1799684592 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 -- 172.21.15.1:0/1799684592 >> v1:172.21.15.3:6789/0 conn(0x7f60bc06c910 legacy=0x7f60bc0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 -- 172.21.15.1:0/1799684592 >> v1:172.21.15.1:6789/0 conn(0x7f60bc15bda0 legacy=0x7f60bc15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60bbfff640 1 -- 172.21.15.1:0/1799684592 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659493734 0 0) 0x7f60bc06ce40 con 0x7f60bc15bda0 2024-03-24T10:34:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.490+0000 7f60c19f2640 1 -- 172.21.15.1:0/1799684592 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc06d400 con 0x7f60bc0d00d0 2024-03-24T10:34:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.491+0000 7f60c19f2640 1 --2- 172.21.15.1:0/1799684592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc166a30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009e50 tx=0x7f60ac02bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c26a8204b364df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.491+0000 7f60bbfff640 1 -- 172.21.15.1:0/1799684592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac02c900 con 0x7f60bc0d00d0 2024-03-24T10:34:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.491+0000 7f60bbfff640 1 -- 172.21.15.1:0/1799684592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac02ca60 con 0x7f60bc0d00d0 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.491+0000 7f60bbfff640 1 -- 172.21.15.1:0/1799684592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac02cdf0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 -- 172.21.15.1:0/1799684592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 msgr2=0x7f60bc166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 --2- 172.21.15.1:0/1799684592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc166a30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009e50 tx=0x7f60ac02bde0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 -- 172.21.15.1:0/1799684592 shutdown_connections 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 --2- 172.21.15.1:0/1799684592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc166a30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 -- 172.21.15.1:0/1799684592 >> 172.21.15.1:0/1799684592 conn(0x7f60bc0b17a0 msgr2=0x7f60bc0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 -- 172.21.15.1:0/1799684592 shutdown_connections 2024-03-24T10:34:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.492+0000 7f60c3455640 1 -- 172.21.15.1:0/1799684592 wait complete. 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.493+0000 7f60c3455640 1 Processor -- start 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.493+0000 7f60c3455640 1 -- start start 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60bc06c910 0x7f60bc14d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60bc15bda0 0x7f60bc1e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc157510 con 0x7f60bc0d00d0 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1577f0 con 0x7f60bc15bda0 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c3455640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60bc151ed0 con 0x7f60bc06c910 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43744/0 (socket says 172.21.15.1:43744) 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 -- 172.21.15.1:0/1547865313 learned_addr learned my addr 172.21.15.1:0/1547865313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c11f1640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60bc06c910 0x7f60bc14d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c19f2640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60bc15bda0 0x7f60bc1e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 -- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60bc06c910 msgr2=0x7f60bc14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60bc06c910 0x7f60bc14d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 -- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60bc15bda0 msgr2=0x7f60bc1e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60bc15bda0 0x7f60bc1e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 -- 172.21.15.1:0/1547865313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac009b30 con 0x7f60bc0d00d0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.494+0000 7f60c09f0640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f60b000c780 tx=0x7f60b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.495+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0005270 con 0x7f60bc0d00d0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.495+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b00053d0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.495+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60bc1521b0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.495+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b000a880 con 0x7f60bc0d00d0 2024-03-24T10:34:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.495+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60bc1567f0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.496+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60bc1523f0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.496+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f60b000a9e0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.497+0000 7f60b9ffb640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 0x7f60a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.497+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f60b0089890 con 0x7f60bc0d00d0 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.497+0000 7f60c11f1640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 0x7f60a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.497+0000 7f60c11f1640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 0x7f60a806dd00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60b4007fd0 tx=0x7f60b4007240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.501+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f60bc1523f0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.683+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60bc1523f0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.684+0000 7f60b9ffb640 1 -- 172.21.15.1:0/1547865313 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16832 (secure 0 0 0) 0x7f60bc1523f0 con 0x7f60bc0d00d0 2024-03-24T10:34:40.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:40.685 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:36.249848+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 msgr2=0x7f60a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 0x7f60a806dd00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60b4007fd0 tx=0x7f60b4007240 comp rx=0 tx=0).stop 2024-03-24T10:34:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 msgr2=0x7f60bc1e40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f60b000c780 tx=0x7f60b000cc40 comp rx=0 tx=0).stop 2024-03-24T10:34:40.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 shutdown_connections 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f60a806b840 0x7f60a806dd00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60bc15bda0 0x7f60bc1e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60bc0d00d0 0x7f60bc1e40e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 --2- 172.21.15.1:0/1547865313 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60bc06c910 0x7f60bc14d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 >> 172.21.15.1:0/1547865313 conn(0x7f60bc0b17a0 msgr2=0x7f60bc159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 shutdown_connections 2024-03-24T10:34:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.686+0000 7f60c3455640 1 -- 172.21.15.1:0/1547865313 wait complete. 2024-03-24T10:34:40.697 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:33:52.583672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:33:53.457579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:40.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:40.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.801+0000 7eff83db5640 1 Processor -- start 2024-03-24T10:34:40.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.801+0000 7eff83db5640 1 -- start start 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff83db5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7eff7c0b4d60 0x7eff7c0d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff83db5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7c0cfef0 con 0x7eff7c161760 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff83db5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7c0b5530 con 0x7eff7c0d0d00 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff83db5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7eff7c0d1170 con 0x7eff7c0b4d60 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff81b51640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7eff7c161760 0x7eff7c0ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46736/0 (socket says 172.21.15.1:46736) 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff81b51640 1 -- 172.21.15.1:0/393421111 learned_addr learned my addr 172.21.15.1:0/393421111 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.802+0000 7eff81350640 1 --2- 172.21.15.1:0/393421111 >> v2:172.21.15.3:3300/0 conn(0x7eff7c0b4d60 0x7eff7c0d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff80b4f640 1 -- 172.21.15.1:0/393421111 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235027347 0 0) 0x7eff7c0cfef0 con 0x7eff7c161760 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff80b4f640 1 -- 172.21.15.1:0/393421111 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff64003520 con 0x7eff7c161760 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff81350640 1 -- 172.21.15.1:0/393421111 >> v1:172.21.15.3:6789/0 conn(0x7eff7c0d0d00 legacy=0x7eff7c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff81350640 1 -- 172.21.15.1:0/393421111 >> v1:172.21.15.1:6789/0 conn(0x7eff7c161760 legacy=0x7eff7c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff81350640 1 -- 172.21.15.1:0/393421111 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c0b6d40 con 0x7eff7c0b4d60 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.803+0000 7eff81350640 1 --2- 172.21.15.1:0/393421111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0b4d60 0x7eff7c0d07c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff700097a0 tx=0x7eff7002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=921082a92ba4a138 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff80b4f640 1 -- 172.21.15.1:0/393421111 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70031830 con 0x7eff7c0b4d60 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff80b4f640 1 -- 172.21.15.1:0/393421111 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff70031990 con 0x7eff7c0b4d60 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff80b4f640 1 -- 172.21.15.1:0/393421111 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70031d20 con 0x7eff7c0b4d60 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 -- 172.21.15.1:0/393421111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0b4d60 msgr2=0x7eff7c0d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 --2- 172.21.15.1:0/393421111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0b4d60 0x7eff7c0d07c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff700097a0 tx=0x7eff7002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 -- 172.21.15.1:0/393421111 shutdown_connections 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 --2- 172.21.15.1:0/393421111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0b4d60 0x7eff7c0d07c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 -- 172.21.15.1:0/393421111 >> 172.21.15.1:0/393421111 conn(0x7eff7c09d1c0 msgr2=0x7eff7c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 -- 172.21.15.1:0/393421111 shutdown_connections 2024-03-24T10:34:40.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.804+0000 7eff83db5640 1 -- 172.21.15.1:0/393421111 wait complete. 2024-03-24T10:34:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.805+0000 7eff83db5640 1 Processor -- start 2024-03-24T10:34:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.805+0000 7eff83db5640 1 -- start start 2024-03-24T10:34:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff7c0b4d60 0x7eff7c0b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0d0d00 0x7eff7c06bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1f4560 con 0x7eff7c161760 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c0b9960 con 0x7eff7c0d0d00 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff83db5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c1582f0 con 0x7eff7c0b4d60 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52570/0 (socket says 172.21.15.1:52570) 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 -- 172.21.15.1:0/1770111266 learned_addr learned my addr 172.21.15.1:0/1770111266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff81350640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0d0d00 0x7eff7c06bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff81b51640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff7c0b4d60 0x7eff7c0b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 -- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff7c0b4d60 msgr2=0x7eff7c0b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff7c0b4d60 0x7eff7c0b9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 -- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0d0d00 msgr2=0x7eff7c06bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0d0d00 0x7eff7c06bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.806+0000 7eff82352640 1 -- 172.21.15.1:0/1770111266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff70009400 con 0x7eff7c161760 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff82352640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c760 tx=0x7eff6c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c006d50 con 0x7eff7c161760 2024-03-24T10:34:40.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c003970 con 0x7eff7c161760 2024-03-24T10:34:40.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c0c8710 con 0x7eff7c161760 2024-03-24T10:34:40.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c003d20 con 0x7eff7c161760 2024-03-24T10:34:40.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.807+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c158870 con 0x7eff7c161760 2024-03-24T10:34:40.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.808+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7c04f4e0 con 0x7eff7c161760 2024-03-24T10:34:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.808+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff6c0050d0 con 0x7eff7c161760 2024-03-24T10:34:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.808+0000 7eff7a7fc640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 0x7eff6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.809+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7eff6c089c90 con 0x7eff7c161760 2024-03-24T10:34:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.809+0000 7eff81b51640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 0x7eff6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:40.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.809+0000 7eff81b51640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 0x7eff6806dcd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff740011e0 tx=0x7eff74007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:40.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.813+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff6c0581c0 con 0x7eff7c161760 2024-03-24T10:34:40.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.992+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff7c153910 con 0x7eff7c161760 2024-03-24T10:34:40.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.993+0000 7eff7a7fc640 1 -- 172.21.15.1:0/1770111266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7eff6c057d50 con 0x7eff7c161760 2024-03-24T10:34:40.995 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:40.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 msgr2=0x7eff6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 0x7eff6806dcd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff740011e0 tx=0x7eff74007d30 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 msgr2=0x7eff7c06c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c760 tx=0x7eff6c00cb60 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 shutdown_connections 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff6806b810 0x7eff6806dcd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.994+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff7c161760 0x7eff7c06c400 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.995+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff7c0d0d00 0x7eff7c06bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.995+0000 7eff83db5640 1 --2- 172.21.15.1:0/1770111266 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff7c0b4d60 0x7eff7c0b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.995+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 >> 172.21.15.1:0/1770111266 conn(0x7eff7c09d1c0 msgr2=0x7eff7c15ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.995+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 shutdown_connections 2024-03-24T10:34:40.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:40.995+0000 7eff83db5640 1 -- 172.21.15.1:0/1770111266 wait complete. 2024-03-24T10:34:41.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:34:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 Processor -- start 2024-03-24T10:34:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 -- start start 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00b31d0 0x7f63c00d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c006d240 0x7f63c00ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c00cc250 con 0x7f63c00d0e90 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f63c00b35b0 con 0x7f63c006d240 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.111+0000 7f63c6ceb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63c00cc4a0 con 0x7f63c00b31d0 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c006d240 0x7f63c00ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63bffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f63c00d0e90 0x7f63c006cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46752/0 (socket says 172.21.15.1:46752) 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63bffff640 1 -- 172.21.15.1:0/1794077730 learned_addr learned my addr 172.21.15.1:0/1794077730 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c006d240 0x7f63c00ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52578/0 (socket says 172.21.15.1:52578) 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c4a87640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00b31d0 0x7f63c00d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 -- 172.21.15.1:0/1794077730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00b31d0 msgr2=0x7f63c00d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00b31d0 0x7f63c00d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 -- 172.21.15.1:0/1794077730 >> v1:172.21.15.1:6789/0 conn(0x7f63c00d0e90 legacy=0x7f63c006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.112+0000 7f63c5288640 1 -- 172.21.15.1:0/1794077730 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c00cc780 con 0x7f63c006d240 2024-03-24T10:34:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c5288640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c006d240 0x7f63c00ce480 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009d90 tx=0x7f63b802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cf47c5a9a670e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63bf7fe640 1 -- 172.21.15.1:0/1794077730 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b802cc50 con 0x7f63c006d240 2024-03-24T10:34:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63bf7fe640 1 -- 172.21.15.1:0/1794077730 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b802cdb0 con 0x7f63c006d240 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63bf7fe640 1 -- 172.21.15.1:0/1794077730 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8034660 con 0x7f63c006d240 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 -- 172.21.15.1:0/1794077730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c006d240 msgr2=0x7f63c00ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c006d240 0x7f63c00ce480 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009d90 tx=0x7f63b802bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 -- 172.21.15.1:0/1794077730 shutdown_connections 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c006d240 0x7f63c00ce480 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/1794077730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00b31d0 0x7f63c00d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 -- 172.21.15.1:0/1794077730 >> 172.21.15.1:0/1794077730 conn(0x7f63c00d1c00 msgr2=0x7f63c00d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 -- 172.21.15.1:0/1794077730 shutdown_connections 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.113+0000 7f63c6ceb640 1 -- 172.21.15.1:0/1794077730 wait complete. 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 Processor -- start 2024-03-24T10:34:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 -- start start 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c006d240 0x7f63c015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00d0e90 0x7f63c00ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f63c00c9490 con 0x7f63c00b31d0 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f63c00c9210 con 0x7f63c006d240 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.115+0000 7f63c6ceb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63c00cb680 con 0x7f63c00d0e90 2024-03-24T10:34:41.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52582/0 (socket says 172.21.15.1:52582) 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 -- 172.21.15.1:0/3620554143 learned_addr learned my addr 172.21.15.1:0/3620554143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63c4a87640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c006d240 0x7f63c015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63c5288640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00d0e90 0x7f63c00ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 -- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00d0e90 msgr2=0x7f63c00ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00d0e90 0x7f63c00ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 -- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c006d240 msgr2=0x7f63c015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c006d240 0x7f63c015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 -- 172.21.15.1:0/3620554143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b8009a70 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bffff640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00c3c0 tx=0x7f63ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ac005ce0 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63ac005e40 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c00cb940 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ac0072d0 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.116+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c00cbf00 con 0x7f63c00b31d0 2024-03-24T10:34:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.117+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c015ebc0 con 0x7f63c00b31d0 2024-03-24T10:34:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.118+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f63ac007430 con 0x7f63c00b31d0 2024-03-24T10:34:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.118+0000 7f63bd7fa640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 0x7f63a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.118+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f63ac0897d0 con 0x7f63c00b31d0 2024-03-24T10:34:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.118+0000 7f63c4a87640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 0x7f63a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.119+0000 7f63c4a87640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 0x7f63a406dd00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009b40 tx=0x7f63b4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.122+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63ac054150 con 0x7f63c00b31d0 2024-03-24T10:34:41.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.305+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f63c004f470 con 0x7f63c00b31d0 2024-03-24T10:34:41.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.305+0000 7f63bd7fa640 1 -- 172.21.15.1:0/3620554143 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f63ac017020 con 0x7f63c00b31d0 2024-03-24T10:34:41.305 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:34:41.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 msgr2=0x7f63a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 0x7f63a406dd00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009b40 tx=0x7f63b4009190 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 msgr2=0x7f63c00c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00c3c0 tx=0x7f63ac00c880 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 shutdown_connections 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a406b840 0x7f63a406dd00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c00d0e90 0x7f63c00ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c00b31d0 0x7f63c00c8cd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 --2- 172.21.15.1:0/3620554143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c006d240 0x7f63c015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 >> 172.21.15.1:0/3620554143 conn(0x7f63c00d1c00 msgr2=0x7f63c006c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 shutdown_connections 2024-03-24T10:34:41.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.307+0000 7f63c6ceb640 1 -- 172.21.15.1:0/3620554143 wait complete. 2024-03-24T10:34:41.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:34:41.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.422+0000 7fddf2e37640 1 Processor -- start 2024-03-24T10:34:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.422+0000 7fddf2e37640 1 -- start start 2024-03-24T10:34:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf2e37640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddec15e390 0x7fddec0d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf2e37640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddec0cfae0 0x7fddec0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf2e37640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec0b0b70 con 0x7fddec0b2ed0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf2e37640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fddec0d1d60 con 0x7fddec15e390 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf2e37640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fddec0b0de0 con 0x7fddec0cfae0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf0bd3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddec15e390 0x7fddec0d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf0bd3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddec15e390 0x7fddec0d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52596/0 (socket says 172.21.15.1:52596) 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fdde3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fddec0b2ed0 0x7fddec0b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46756/0 (socket says 172.21.15.1:46756) 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf0bd3640 1 -- 172.21.15.1:0/490281498 learned_addr learned my addr 172.21.15.1:0/490281498 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.423+0000 7fddf13d4640 1 --2- 172.21.15.1:0/490281498 >> v2:172.21.15.3:3300/0 conn(0x7fddec0cfae0 0x7fddec0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fddf0bd3640 1 -- 172.21.15.1:0/490281498 >> v2:172.21.15.3:3300/0 conn(0x7fddec0cfae0 msgr2=0x7fddec0d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fddf0bd3640 1 --2- 172.21.15.1:0/490281498 >> v2:172.21.15.3:3300/0 conn(0x7fddec0cfae0 0x7fddec0d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fddf0bd3640 1 -- 172.21.15.1:0/490281498 >> v1:172.21.15.1:6789/0 conn(0x7fddec0b2ed0 legacy=0x7fddec0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fddf0bd3640 1 -- 172.21.15.1:0/490281498 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec15e770 con 0x7fddec15e390 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/490281498 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118467189 0 0) 0x7fddec0b0b70 con 0x7fddec0b2ed0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fddf0bd3640 1 --2- 172.21.15.1:0/490281498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0d2310 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fddec08d820 tx=0x7fdde802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa5526f7efbb480 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.424+0000 7fdde37fe640 1 -- 172.21.15.1:0/490281498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8030860 con 0x7fddec15e390 2024-03-24T10:34:41.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/490281498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde80309c0 con 0x7fddec15e390 2024-03-24T10:34:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fdde37fe640 1 -- 172.21.15.1:0/490281498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8030d50 con 0x7fddec15e390 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 -- 172.21.15.1:0/490281498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 msgr2=0x7fddec0d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 --2- 172.21.15.1:0/490281498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0d2310 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fddec08d820 tx=0x7fdde802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 -- 172.21.15.1:0/490281498 shutdown_connections 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 --2- 172.21.15.1:0/490281498 >> v2:172.21.15.3:3300/0 conn(0x7fddec0cfae0 0x7fddec0d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 --2- 172.21.15.1:0/490281498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0d2310 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 -- 172.21.15.1:0/490281498 >> 172.21.15.1:0/490281498 conn(0x7fddec0d3ee0 msgr2=0x7fddec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 -- 172.21.15.1:0/490281498 shutdown_connections 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.425+0000 7fddf2e37640 1 -- 172.21.15.1:0/490281498 wait complete. 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.426+0000 7fddf2e37640 1 Processor -- start 2024-03-24T10:34:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.426+0000 7fddf2e37640 1 -- start start 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddec0b2ed0 0x7fddec1f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fddec0cfae0 0x7fddec1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fddec1f4e90 con 0x7fddec15e390 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fddec0b6bf0 con 0x7fddec0b2ed0 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf2e37640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fddec0b6ed0 con 0x7fddec0cfae0 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf13d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf13d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52602/0 (socket says 172.21.15.1:52602) 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf13d4640 1 -- 172.21.15.1:0/3674328927 learned_addr learned my addr 172.21.15.1:0/3674328927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fdde3fff640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fddec0cfae0 0x7fddec1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.427+0000 7fddf0bd3640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddec0b2ed0 0x7fddec1f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 -- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fddec0cfae0 msgr2=0x7fddec1f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fddec0cfae0 0x7fddec1f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 -- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddec0b2ed0 msgr2=0x7fddec1f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddec0b2ed0 0x7fddec1f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 -- 172.21.15.1:0/3674328927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde80094d0 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf13d4640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdde400efc0 tx=0x7fdde400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4014480 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde400f040 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec0b71b0 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4014820 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.428+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec0b7770 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.429+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddec0b7960 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.429+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdde4014980 con 0x7fddec15e390 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.430+0000 7fdde17fa640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 0x7fddd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.430+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fdde408acc0 con 0x7fddec15e390 2024-03-24T10:34:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.430+0000 7fddf0bd3640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 0x7fddd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.430+0000 7fddf0bd3640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 0x7fddd806dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdde8036000 tx=0x7fdde8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.434+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdde4055760 con 0x7fddec15e390 2024-03-24T10:34:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.615+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fddec04f4e0 con 0x7fddec15e390 2024-03-24T10:34:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.615+0000 7fdde17fa640 1 -- 172.21.15.1:0/3674328927 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7fdde4018020 con 0x7fddec15e390 2024-03-24T10:34:41.615 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:34:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 msgr2=0x7fddd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 0x7fddd806dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdde8036000 tx=0x7fdde8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 msgr2=0x7fddec0b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdde400efc0 tx=0x7fdde400c490 comp rx=0 tx=0).stop 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 shutdown_connections 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddd806b810 0x7fddd806dcd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddec15e390 0x7fddec0b3d20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fddec0cfae0 0x7fddec1f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 --2- 172.21.15.1:0/3674328927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddec0b2ed0 0x7fddec1f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 >> 172.21.15.1:0/3674328927 conn(0x7fddec0d3ee0 msgr2=0x7fddec0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 shutdown_connections 2024-03-24T10:34:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.617+0000 7fddf2e37640 1 -- 172.21.15.1:0/3674328927 wait complete. 2024-03-24T10:34:41.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:41.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.730+0000 7fa049e15640 1 Processor -- start 2024-03-24T10:34:41.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.730+0000 7fa049e15640 1 -- start start 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa049e15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0440cd830 0x7fa0440cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa049e15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440d27a0 0x7fa0440b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa049e15640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0440cc790 con 0x7fa0440cfd00 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa049e15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa0440d1930 con 0x7fa0440cd830 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa049e15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa0440d0be0 con 0x7fa0440d27a0 2024-03-24T10:34:41.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa042ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0440cd830 0x7fa0440cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa0437fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0440cfd00 0x7fa0440d00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46758/0 (socket says 172.21.15.1:46758) 2024-03-24T10:34:41.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa0437fe640 1 -- 172.21.15.1:0/2000459156 learned_addr learned my addr 172.21.15.1:0/2000459156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa043fff640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440d27a0 0x7fa0440b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa042ffd640 1 -- 172.21.15.1:0/2000459156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440d27a0 msgr2=0x7fa0440b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa042ffd640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440d27a0 0x7fa0440b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa042ffd640 1 -- 172.21.15.1:0/2000459156 >> v1:172.21.15.1:6789/0 conn(0x7fa0440cfd00 legacy=0x7fa0440d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa042ffd640 1 -- 172.21.15.1:0/2000459156 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0440cca70 con 0x7fa0440cd830 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.731+0000 7fa0427fc640 1 -- 172.21.15.1:0/2000459156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484797188 0 0) 0x7fa0440cc790 con 0x7fa0440cfd00 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.732+0000 7fa042ffd640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440cf0a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa0380097a0 tx=0x7fa03802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a228e1954b432798 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.732+0000 7fa0427fc640 1 -- 172.21.15.1:0/2000459156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03802e070 con 0x7fa0440cd830 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.732+0000 7fa0427fc640 1 -- 172.21.15.1:0/2000459156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0380315a0 con 0x7fa0440cd830 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.732+0000 7fa0427fc640 1 -- 172.21.15.1:0/2000459156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038031930 con 0x7fa0440cd830 2024-03-24T10:34:41.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 -- 172.21.15.1:0/2000459156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 msgr2=0x7fa0440cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440cf0a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa0380097a0 tx=0x7fa03802b840 comp rx=0 tx=0).stop 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 -- 172.21.15.1:0/2000459156 shutdown_connections 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440d27a0 0x7fa0440b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 --2- 172.21.15.1:0/2000459156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440cf0a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 -- 172.21.15.1:0/2000459156 >> 172.21.15.1:0/2000459156 conn(0x7fa0440d1c80 msgr2=0x7fa0440d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 -- 172.21.15.1:0/2000459156 shutdown_connections 2024-03-24T10:34:41.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.733+0000 7fa049e15640 1 -- 172.21.15.1:0/2000459156 wait complete. 2024-03-24T10:34:41.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.734+0000 7fa049e15640 1 Processor -- start 2024-03-24T10:34:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.734+0000 7fa049e15640 1 -- start start 2024-03-24T10:34:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440cfd00 0x7fa0441f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440d27a0 0x7fa0441f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa0440b7610 con 0x7fa0440cd830 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa0440b6a00 con 0x7fa0440d27a0 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa0440b6ce0 con 0x7fa0440cfd00 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52612/0 (socket says 172.21.15.1:52612) 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 -- 172.21.15.1:0/1280273043 learned_addr learned my addr 172.21.15.1:0/1280273043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa042ffd640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440cfd00 0x7fa0441f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa043fff640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440d27a0 0x7fa0441f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 -- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440cfd00 msgr2=0x7fa0441f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440cfd00 0x7fa0441f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 -- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440d27a0 msgr2=0x7fa0441f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440d27a0 0x7fa0441f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 -- 172.21.15.1:0/1280273043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa038009400 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa0437fe640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa03400c430 tx=0x7fa03400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0340074e0 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa034007660 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0440b6fc0 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034004a20 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.735+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0440b5010 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.737+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04404f4e0 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.737+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa034004b80 con 0x7fa0440cd830 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.737+0000 7fa02bfff640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 0x7fa03006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:41.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.737+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fa034089f50 con 0x7fa0440cd830 2024-03-24T10:34:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.737+0000 7fa042ffd640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 0x7fa03006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.738+0000 7fa042ffd640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 0x7fa03006dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa038036000 tx=0x7fa03803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:41.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.742+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa034054960 con 0x7fa0440cd830 2024-03-24T10:34:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.962+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0440ce470 con 0x7fa0440cd830 2024-03-24T10:34:41.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.962+0000 7fa02bfff640 1 -- 172.21.15.1:0/1280273043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7fa034058020 con 0x7fa0440cd830 2024-03-24T10:34:41.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:41.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:41.963 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 msgr2=0x7fa03006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 0x7fa03006dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa038036000 tx=0x7fa03803d040 comp rx=0 tx=0).stop 2024-03-24T10:34:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 msgr2=0x7fa0440b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa03400c430 tx=0x7fa03400c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 shutdown_connections 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa03006b810 0x7fa03006dcd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440d27a0 0x7fa0441f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0440cfd00 0x7fa0441f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 --2- 172.21.15.1:0/1280273043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0440cd830 0x7fa0440b4ad0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 >> 172.21.15.1:0/1280273043 conn(0x7fa0440d1c80 msgr2=0x7fa04415ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 shutdown_connections 2024-03-24T10:34:41.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:41.964+0000 7fa049e15640 1 -- 172.21.15.1:0/1280273043 wait complete. 2024-03-24T10:34:41.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:42.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.077+0000 7f4f4373e640 1 Processor -- start 2024-03-24T10:34:42.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.077+0000 7f4f4373e640 1 -- start start 2024-03-24T10:34:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f4373e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f4373e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4f3c0b2aa0 0x7f4f3c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f4373e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3c0cdd60 con 0x7f4f3c144770 2024-03-24T10:34:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f4373e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3c06eaa0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f4373e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c0b3470 con 0x7f4f3c0d00f0 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f414da640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4f3c144770 0x7f4f3c06da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46768/0 (socket says 172.21.15.1:46768) 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f414da640 1 -- 172.21.15.1:0/1056574824 learned_addr learned my addr 172.21.15.1:0/1056574824 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f41cdb640 1 --2- 172.21.15.1:0/1056574824 >> v2:172.21.15.3:3300/0 conn(0x7f4f3c0b2aa0 0x7f4f3c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f40cd9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f41cdb640 1 -- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 msgr2=0x7f4f3c070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f41cdb640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c070b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f33fff640 1 -- 172.21.15.1:0/1056574824 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2787764842 0 0) 0x7f4f3c0cdd60 con 0x7f4f3c144770 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f41cdb640 1 -- 172.21.15.1:0/1056574824 >> v1:172.21.15.1:6789/0 conn(0x7f4f3c144770 legacy=0x7f4f3c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f41cdb640 1 -- 172.21.15.1:0/1056574824 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c06e200 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.078+0000 7f4f40cd9640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f41cdb640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c0b2f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009d90 tx=0x7f4f3402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4479e579abaace75 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f33fff640 1 -- 172.21.15.1:0/1056574824 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3402cc50 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f33fff640 1 -- 172.21.15.1:0/1056574824 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3402cdb0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f33fff640 1 -- 172.21.15.1:0/1056574824 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34034660 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f4373e640 1 -- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c0b2aa0 msgr2=0x7f4f3c0b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.079+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c0b2f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009d90 tx=0x7f4f3402bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 -- 172.21.15.1:0/1056574824 shutdown_connections 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c0b2f30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1056574824 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 -- 172.21.15.1:0/1056574824 >> 172.21.15.1:0/1056574824 conn(0x7f4f3c0b17a0 msgr2=0x7f4f3c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 -- 172.21.15.1:0/1056574824 shutdown_connections 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.080+0000 7f4f4373e640 1 -- 172.21.15.1:0/1056574824 wait complete. 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.081+0000 7f4f4373e640 1 Processor -- start 2024-03-24T10:34:42.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.081+0000 7f4f4373e640 1 -- start start 2024-03-24T10:34:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.081+0000 7f4f4373e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c13ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c144770 0x7f4f3c1f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c0aa1a0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c13dca0 con 0x7f4f3c144770 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c13df80 con 0x7f4f3c0d00f0 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52624/0 (socket says 172.21.15.1:52624) 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 -- 172.21.15.1:0/1049714358 learned_addr learned my addr 172.21.15.1:0/1049714358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f40cd9640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c13ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f41cdb640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c144770 0x7f4f3c1f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 -- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 msgr2=0x7f4f3c13ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c13ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 -- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c144770 msgr2=0x7f4f3c1f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c144770 0x7f4f3c1f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 -- 172.21.15.1:0/1049714358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c009400 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f414da640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c430 tx=0x7f4f3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f380074e0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f38007660 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f34009a70 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38004a20 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.082+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c13e540 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.083+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3c04f4e0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.084+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f38004b80 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.084+0000 7f4f31ffb640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 0x7f4f2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.084+0000 7f4f40cd9640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 0x7f4f2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.084+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f4f38089ef0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.085+0000 7f4f40cd9640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 0x7f4f2006dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c13c030 tx=0x7f4f2c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.089+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f38054990 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.271+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3c15e1d0 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.272+0000 7f4f31ffb640 1 -- 172.21.15.1:0/1049714358 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16832 (secure 0 0 0) 0x7f4f38058050 con 0x7f4f3c0b2aa0 2024-03-24T10:34:42.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:42.273 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:36.249848+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.274+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 msgr2=0x7f4f2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 0x7f4f2006dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c13c030 tx=0x7f4f2c009190 comp rx=0 tx=0).stop 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 msgr2=0x7f4f3c15a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c430 tx=0x7f4f3800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 shutdown_connections 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f2006b810 0x7f4f2006dcd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f3c144770 0x7f4f3c1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4f3c0d00f0 0x7f4f3c13ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 --2- 172.21.15.1:0/1049714358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f3c0b2aa0 0x7f4f3c15a4c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 >> 172.21.15.1:0/1049714358 conn(0x7f4f3c0b17a0 msgr2=0x7f4f3c15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 shutdown_connections 2024-03-24T10:34:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.275+0000 7f4f4373e640 1 -- 172.21.15.1:0/1049714358 wait complete. 2024-03-24T10:34:42.286 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T10:34:42.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.388+0000 7f16ba09d640 1 Processor -- start 2024-03-24T10:34:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.388+0000 7f16ba09d640 1 -- start start 2024-03-24T10:34:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b415e0a0 0x7f16b406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f16b4072710 0x7f16b406b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16b40b0180 0x7f16b4161c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f16b4072c10 con 0x7f16b40b0180 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f16b406dde0 con 0x7f16b4072710 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16ba09d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f16b406e0c0 con 0x7f16b415e0a0 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16b40b0180 0x7f16b4161c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b415e0a0 0x7f16b406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16b40b0180 0x7f16b4161c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52636/0 (socket says 172.21.15.1:52636) 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 -- 172.21.15.1:0/1924809582 learned_addr learned my addr 172.21.15.1:0/1924809582 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b2ffd640 1 --2- 172.21.15.1:0/1924809582 >> v2:172.21.15.3:3300/0 conn(0x7f16b4072710 0x7f16b406b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 -- 172.21.15.1:0/1924809582 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b415e0a0 msgr2=0x7f16b406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 --2- 172.21.15.1:0/1924809582 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b415e0a0 0x7f16b406b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 -- 172.21.15.1:0/1924809582 >> v2:172.21.15.3:3300/0 conn(0x7f16b4072710 msgr2=0x7f16b406b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 --2- 172.21.15.1:0/1924809582 >> v2:172.21.15.3:3300/0 conn(0x7f16b4072710 0x7f16b406b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.389+0000 7f16b3fff640 1 -- 172.21.15.1:0/1924809582 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b406e3a0 con 0x7f16b40b0180 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.390+0000 7f16b3fff640 1 --2- 172.21.15.1:0/1924809582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b40b0180 0x7f16b4161c70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16b406d060 tx=0x7f16ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec402be8a1086f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.390+0000 7f16b27fc640 1 -- 172.21.15.1:0/1924809582 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac02ca70 con 0x7f16b40b0180 2024-03-24T10:34:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.390+0000 7f16b27fc640 1 -- 172.21.15.1:0/1924809582 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac02cbd0 con 0x7f16b40b0180 2024-03-24T10:34:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.390+0000 7f16b27fc640 1 -- 172.21.15.1:0/1924809582 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0343f0 con 0x7f16b40b0180 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 -- 172.21.15.1:0/1924809582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b40b0180 msgr2=0x7f16b4161c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 --2- 172.21.15.1:0/1924809582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b40b0180 0x7f16b4161c70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16b406d060 tx=0x7f16ac02bed0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 -- 172.21.15.1:0/1924809582 shutdown_connections 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 --2- 172.21.15.1:0/1924809582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b40b0180 0x7f16b4161c70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 --2- 172.21.15.1:0/1924809582 >> v2:172.21.15.3:3300/0 conn(0x7f16b4072710 0x7f16b406b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 --2- 172.21.15.1:0/1924809582 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b415e0a0 0x7f16b406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 -- 172.21.15.1:0/1924809582 >> 172.21.15.1:0/1924809582 conn(0x7f16b40b2820 msgr2=0x7f16b40b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 -- 172.21.15.1:0/1924809582 shutdown_connections 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.391+0000 7f16ba09d640 1 -- 172.21.15.1:0/1924809582 wait complete. 2024-03-24T10:34:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.392+0000 7f16ba09d640 1 Processor -- start 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.392+0000 7f16ba09d640 1 -- start start 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b40b0180 0x7f16b4160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16b415e0a0 0x7f16b414c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40c6210 con 0x7f16b4072710 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f16b414bbe0 con 0x7f16b415e0a0 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16ba09d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f16b4161630 con 0x7f16b40b0180 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52642/0 (socket says 172.21.15.1:52642) 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 -- 172.21.15.1:0/2617653356 learned_addr learned my addr 172.21.15.1:0/2617653356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b2ffd640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b40b0180 0x7f16b4160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b3fff640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16b415e0a0 0x7f16b414c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 -- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b40b0180 msgr2=0x7f16b4160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b40b0180 0x7f16b4160970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 -- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16b415e0a0 msgr2=0x7f16b414c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16b415e0a0 0x7f16b414c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.393+0000 7f16b37fe640 1 -- 172.21.15.1:0/2617653356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac009b30 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f16b37fe640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16a400e8a0 tx=0x7f16a400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40094f0 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a400c940 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b4161350 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a400ccf0 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.394+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b40c5ea0 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.395+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b404f4e0 con 0x7f16b4072710 2024-03-24T10:34:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.395+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f16a4013470 con 0x7f16b4072710 2024-03-24T10:34:42.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.395+0000 7f169bfff640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 0x7f16a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.396+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f16a408a250 con 0x7f16b4072710 2024-03-24T10:34:42.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.396+0000 7f16b2ffd640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 0x7f16a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.396+0000 7f16b2ffd640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 0x7f16a006dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f16b40c73d0 tx=0x7f16a8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.400+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f16a4054cf0 con 0x7f16b4072710 2024-03-24T10:34:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.617+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16b40b3f30 con 0x7f16b4072710 2024-03-24T10:34:42.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.618+0000 7f169bfff640 1 -- 172.21.15.1:0/2617653356 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f16a40583b0 con 0x7f16b4072710 2024-03-24T10:34:42.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:42.619 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.619+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 msgr2=0x7f16a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.619+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 0x7f16a006dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f16b40c73d0 tx=0x7f16a8009190 comp rx=0 tx=0).stop 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.619+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 msgr2=0x7f16b414b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.619+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16a400e8a0 tx=0x7f16a400ed60 comp rx=0 tx=0).stop 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.619+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 shutdown_connections 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f16a006b810 0x7f16a006dcd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16b415e0a0 0x7f16b414c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f16b40b0180 0x7f16b4160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 --2- 172.21.15.1:0/2617653356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16b4072710 0x7f16b414b6a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 >> 172.21.15.1:0/2617653356 conn(0x7f16b40b2820 msgr2=0x7f16b40af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 shutdown_connections 2024-03-24T10:34:42.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.620+0000 7f16ba09d640 1 -- 172.21.15.1:0/2617653356 wait complete. 2024-03-24T10:34:42.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.732+0000 7f55c7bb5640 1 Processor -- start 2024-03-24T10:34:42.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.732+0000 7f55c7bb5640 1 -- start start 2024-03-24T10:34:42.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c7bb5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 0x7f55c00cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c7bb5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55c00cff00 0x7f55c0165a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c7bb5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55c006ce40 con 0x7f55c015bda0 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c7bb5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f55c00b2db0 con 0x7f55c00cff00 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c7bb5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f55c006e140 con 0x7f55c006c910 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c5150640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 0x7f55c00cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55c00cff00 0x7f55c0165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55c00cff00 0x7f55c0165a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59716/0 (socket says 172.21.15.1:59716) 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c5951640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f55c015bda0 0x7f55c015ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:58956/0 (socket says 172.21.15.1:58956) 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c5150640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 0x7f55c00cf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:42592/0 (socket says 172.21.15.1:42592) 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 -- 172.21.15.1:0/3645590012 learned_addr learned my addr 172.21.15.1:0/3645590012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 -- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 msgr2=0x7f55c00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 --2- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 0x7f55c00cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 -- 172.21.15.1:0/3645590012 >> v1:172.21.15.3:6789/0 conn(0x7f55c015bda0 legacy=0x7f55c015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c494f640 1 -- 172.21.15.1:0/3645590012 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3172183248 0 0) 0x7f55c006ce40 con 0x7f55c015bda0 2024-03-24T10:34:42.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.733+0000 7f55c6152640 1 -- 172.21.15.1:0/3645590012 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c006d120 con 0x7f55c00cff00 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.734+0000 7f55c6152640 1 --2- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c00cff00 0x7f55c0165a20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab54af5c5c39c32c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.734+0000 7f55c494f640 1 -- 172.21.15.1:0/3645590012 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b802ca70 con 0x7f55c00cff00 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.734+0000 7f55c494f640 1 -- 172.21.15.1:0/3645590012 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b802cbd0 con 0x7f55c00cff00 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.734+0000 7f55c494f640 1 -- 172.21.15.1:0/3645590012 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b80343f0 con 0x7f55c00cff00 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c00cff00 msgr2=0x7f55c0165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c00cff00 0x7f55c0165a20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009e50 tx=0x7f55b802bed0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3645590012 shutdown_connections 2024-03-24T10:34:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c00cff00 0x7f55c0165a20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3645590012 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c006c910 0x7f55c00cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3645590012 >> 172.21.15.1:0/3645590012 conn(0x7f55c00b17a0 msgr2=0x7f55c00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3645590012 shutdown_connections 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.735+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3645590012 wait complete. 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.736+0000 7f55c7bb5640 1 Processor -- start 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.736+0000 7f55c7bb5640 1 -- start start 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c00cff00 0x7f55c014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 0x7f55c014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f55c013d220 con 0x7f55c015bda0 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f55c014dc00 con 0x7f55c006c910 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c7bb5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f55c014dee0 con 0x7f55c00cff00 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c6152640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 0x7f55c014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59718/0 (socket says 172.21.15.1:59718) 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 -- 172.21.15.1:0/3405552082 learned_addr learned my addr 172.21.15.1:0/3405552082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5150640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c00cff00 0x7f55c014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 -- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c00cff00 msgr2=0x7f55c014e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c00cff00 0x7f55c014e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 -- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 msgr2=0x7f55c014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 0x7f55c014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c5951640 1 -- 172.21.15.1:0/3405552082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b8009b30 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.737+0000 7f55c6152640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 0x7f55c014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55c5951640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00c430 tx=0x7f55bc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0074e0 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55bc007660 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c013cbd0 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc004a20 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.738+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c0140db0 con 0x7f55c006c910 2024-03-24T10:34:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.739+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c004f4e0 con 0x7f55c006c910 2024-03-24T10:34:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.739+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f55bc004b80 con 0x7f55c006c910 2024-03-24T10:34:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.739+0000 7f55b67fc640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 0x7f55a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.739+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f55bc089f70 con 0x7f55c006c910 2024-03-24T10:34:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.740+0000 7f55c5150640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 0x7f55a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.740+0000 7f55c5150640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 0x7f55a406dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55c014d0f0 tx=0x7f55b0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:42.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.744+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f55bc0bde10 con 0x7f55c006c910 2024-03-24T10:34:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.969+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55c0072b80 con 0x7f55c006c910 2024-03-24T10:34:42.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.970+0000 7f55b67fc640 1 -- 172.21.15.1:0/3405552082 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5157 (secure 0 0 0) 0x7f55bc054980 con 0x7f55c006c910 2024-03-24T10:34:42.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:42.970 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/4196687649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":4196687649},{"type":"v1","addr":"172.21.15.3:6835","nonce":4196687649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/612363354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":612363354},{"type":"v1","addr":"172.21.15.1:6835","nonce":612363354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4123993138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4123993138},{"type":"v1","addr":"172.21.15.1:6837","nonce":4123993138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:33:54.483199+0000","modified":"2024-03-24T10:33:57.542563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6834/824939987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":824939987},{"type":"v1","addr":"172.21.15.3:6834","nonce":824939987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-03-24T10:34:42.971 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-03-24T10:34:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 msgr2=0x7f55a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 0x7f55a406dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55c014d0f0 tx=0x7f55b0009190 comp rx=0 tx=0).stop 2024-03-24T10:34:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 msgr2=0x7f55c013d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00c430 tx=0x7f55bc00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 shutdown_connections 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f55a406b810 0x7f55a406dcd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55c015bda0 0x7f55c014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f55c00cff00 0x7f55c014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 --2- 172.21.15.1:0/3405552082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55c006c910 0x7f55c013d6b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 >> 172.21.15.1:0/3405552082 conn(0x7f55c00b17a0 msgr2=0x7f55c00b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 shutdown_connections 2024-03-24T10:34:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:42.972+0000 7f55c7bb5640 1 -- 172.21.15.1:0/3405552082 wait complete. 2024-03-24T10:34:42.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.083+0000 7f799270c640 1 Processor -- start 2024-03-24T10:34:43.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.083+0000 7f799270c640 1 -- start start 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f799270c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f798c0d2bb0 0x7f798c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f799270c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d1000 0x7f798c0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f799270c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f798c15e770 con 0x7f798c15e390 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f799270c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f798c0b3190 con 0x7f798c0d2bb0 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f799270c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f798c0d4e40 con 0x7f798c0d1000 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f798b7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f798c0d2bb0 0x7f798c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f798b7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f798c0d2bb0 0x7f798c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59730/0 (socket says 172.21.15.1:59730) 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f798b7fe640 1 -- 172.21.15.1:0/3055772334 learned_addr learned my addr 172.21.15.1:0/3055772334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.084+0000 7f7990ca9640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d1000 0x7f798c0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.085+0000 7f798b7fe640 1 -- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d1000 msgr2=0x7f798c0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.085+0000 7f798b7fe640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d1000 0x7f798c0b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.085+0000 7f798b7fe640 1 -- 172.21.15.1:0/3055772334 >> v1:172.21.15.3:6789/0 conn(0x7f798c15e390 legacy=0x7f798c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.085+0000 7f798affd640 1 -- 172.21.15.1:0/3055772334 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312470118 0 0) 0x7f797c001110 con 0x7f798c15e390 2024-03-24T10:34:43.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.085+0000 7f798b7fe640 1 -- 172.21.15.1:0/3055772334 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c0d1e40 con 0x7f798c0d2bb0 2024-03-24T10:34:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f798b7fe640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d2bb0 0x7f798c0b3b90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad37270b4b1b47e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f798affd640 1 -- 172.21.15.1:0/3055772334 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79800317c0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f798affd640 1 -- 172.21.15.1:0/3055772334 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7980031920 con 0x7f798c0d2bb0 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f798affd640 1 -- 172.21.15.1:0/3055772334 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980031cb0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 -- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d2bb0 msgr2=0x7f798c0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d2bb0 0x7f798c0b3b90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b860 comp rx=0 tx=0).stop 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 -- 172.21.15.1:0/3055772334 shutdown_connections 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d1000 0x7f798c0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 --2- 172.21.15.1:0/3055772334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d2bb0 0x7f798c0b3b90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 -- 172.21.15.1:0/3055772334 >> 172.21.15.1:0/3055772334 conn(0x7f798c0d3ec0 msgr2=0x7f798c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 -- 172.21.15.1:0/3055772334 shutdown_connections 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.086+0000 7f799270c640 1 -- 172.21.15.1:0/3055772334 wait complete. 2024-03-24T10:34:43.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.087+0000 7f799270c640 1 Processor -- start 2024-03-24T10:34:43.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 -- start start 2024-03-24T10:34:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 0x7f798c15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 0x7f798c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f798c0caf90 con 0x7f798c15e390 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f798c0cbb10 con 0x7f798c0d1000 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f799270c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f798c155920 con 0x7f798c0d2bb0 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f7990ca9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f798bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 0x7f798c15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f7990ca9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52658/0 (socket says 172.21.15.1:52658) 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f7990ca9640 1 -- 172.21.15.1:0/1449723579 learned_addr learned my addr 172.21.15.1:0/1449723579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.088+0000 7f798b7fe640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 0x7f798c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 -- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 msgr2=0x7f798c15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 0x7f798c15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 -- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 msgr2=0x7f798c1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 -- 172.21.15.1:0/1449723579 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79800094d0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798bfff640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 0x7f798c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f7990ca9640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f798b7fe640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 0x7f798c1553e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7980033000 tx=0x7f79800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980004220 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7980031f00 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798c0cb1e0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980003610 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.089+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f798c0cb7a0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.090+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c0cbc80 con 0x7f798c0d2bb0 2024-03-24T10:34:43.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.091+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7980043070 con 0x7f798c0d2bb0 2024-03-24T10:34:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.091+0000 7f7988ff9640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 0x7f797806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.091+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7f79800abfc0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.091+0000 7f798bfff640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 0x7f797806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.092+0000 7f798bfff640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 0x7f797806dd00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f797c000f60 tx=0x7f797c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.100+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7980076a60 con 0x7f798c0d2bb0 2024-03-24T10:34:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.284+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f798c04f4e0 con 0x7f798c0d2bb0 2024-03-24T10:34:43.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.286+0000 7f7988ff9640 1 -- 172.21.15.1:0/1449723579 <== mon.2 v2:172.21.15.3:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16832 (secure 0 0 0) 0x7f798007a120 con 0x7f798c0d2bb0 2024-03-24T10:34:43.286 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:43.286 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:36.249848+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:33:52.583672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:33:53.457579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 msgr2=0x7f797806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 0x7f797806dd00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f797c000f60 tx=0x7f797c009040 comp rx=0 tx=0).stop 2024-03-24T10:34:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 msgr2=0x7f798c1553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 0x7f798c1553e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7980033000 tx=0x7f79800046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 shutdown_connections 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f797806b840 0x7f797806dd00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15e390 0x7f798c1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f798c0d2bb0 0x7f798c1553e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 --2- 172.21.15.1:0/1449723579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f798c0d1000 0x7f798c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 >> 172.21.15.1:0/1449723579 conn(0x7f798c0d3ec0 msgr2=0x7f798c0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 shutdown_connections 2024-03-24T10:34:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.288+0000 7f799270c640 1 -- 172.21.15.1:0/1449723579 wait complete. 2024-03-24T10:34:43.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T10:34:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 Processor -- start 2024-03-24T10:34:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 -- start start 2024-03-24T10:34:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe900d2bb0 0x7fbe900b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d1000 0x7fbe900b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9015e770 con 0x7fbe9015e390 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbe900b3190 con 0x7fbe900d2bb0 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.402+0000 7fbe9684d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbe900d4e40 con 0x7fbe900d1000 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe900d2bb0 0x7fbe900b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe900d2bb0 0x7fbe900b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52672/0 (socket says 172.21.15.1:52672) 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 -- 172.21.15.1:0/2503731275 learned_addr learned my addr 172.21.15.1:0/2503731275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe94dea640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d1000 0x7fbe900b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 -- 172.21.15.1:0/2503731275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d1000 msgr2=0x7fbe900b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d1000 0x7fbe900b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8effd640 1 -- 172.21.15.1:0/2503731275 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 127983360 0 0) 0x7fbe80001110 con 0x7fbe9015e390 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 -- 172.21.15.1:0/2503731275 >> v1:172.21.15.3:6789/0 conn(0x7fbe9015e390 legacy=0x7fbe900d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.403+0000 7fbe8f7fe640 1 -- 172.21.15.1:0/2503731275 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900d1e40 con 0x7fbe900d2bb0 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe8f7fe640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe900d2bb0 0x7fbe900b3b90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009870 tx=0x7fbe8402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f930204c28744d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe8effd640 1 -- 172.21.15.1:0/2503731275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe840317c0 con 0x7fbe900d2bb0 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe8effd640 1 -- 172.21.15.1:0/2503731275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe84031920 con 0x7fbe900d2bb0 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe8effd640 1 -- 172.21.15.1:0/2503731275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84031cb0 con 0x7fbe900d2bb0 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 -- 172.21.15.1:0/2503731275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe900d2bb0 msgr2=0x7fbe900b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe900d2bb0 0x7fbe900b3b90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009870 tx=0x7fbe8402b860 comp rx=0 tx=0).stop 2024-03-24T10:34:43.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 -- 172.21.15.1:0/2503731275 shutdown_connections 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d1000 0x7fbe900b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2503731275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe900d2bb0 0x7fbe900b3b90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 -- 172.21.15.1:0/2503731275 >> 172.21.15.1:0/2503731275 conn(0x7fbe900d3ec0 msgr2=0x7fbe900d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 -- 172.21.15.1:0/2503731275 shutdown_connections 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.404+0000 7fbe9684d640 1 -- 172.21.15.1:0/2503731275 wait complete. 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.405+0000 7fbe9684d640 1 Processor -- start 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 -- start start 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe900d1000 0x7fbe901515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d2bb0 0x7fbe90151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900caf90 con 0x7fbe9015e390 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900ccca0 con 0x7fbe900d1000 2024-03-24T10:34:43.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.406+0000 7fbe9684d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbe901521f0 con 0x7fbe900d2bb0 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52674/0 (socket says 172.21.15.1:52674) 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe900d1000 0x7fbe901515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 -- 172.21.15.1:0/2459849791 learned_addr learned my addr 172.21.15.1:0/2459849791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe8f7fe640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d2bb0 0x7fbe90151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 -- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d2bb0 msgr2=0x7fbe90151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d2bb0 0x7fbe90151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 -- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe900d1000 msgr2=0x7fbe901515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe900d1000 0x7fbe901515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 -- 172.21.15.1:0/2459849791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe840094d0 con 0x7fbe9015e390 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe94dea640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800efc0 tx=0x7fbe8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88009ca0 con 0x7fbe9015e390 2024-03-24T10:34:43.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8800f040 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901524d0 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880146c0 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.407+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe90152a90 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.408+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe900cce10 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.409+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbe88014820 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.409+0000 7fbe8cff9640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.409+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5891+0+0 (secure 0 0 0) 0x7fbe8808b0a0 con 0x7fbe9015e390 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.409+0000 7fbe8ffff640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.410+0000 7fbe8ffff640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000f60 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.413+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbe900cce10 con 0x7fbe9015e390 2024-03-24T10:34:43.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.631+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbe900cce10 con 0x7fbe9015e390 2024-03-24T10:34:43.632 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-24T10:34:43.632+0000 7f388d532640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:34:43.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.633+0000 7fbe8cff9640 1 -- 172.21.15.1:0/2459849791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fbe900cce10 con 0x7fbe9015e390 2024-03-24T10:34:43.633 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.636+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 msgr2=0x7fbe7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.636+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000f60 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.636+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 msgr2=0x7fbe900a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.636+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800efc0 tx=0x7fbe8800c490 comp rx=0 tx=0).stop 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 shutdown_connections 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe9015e390 0x7fbe900a0cb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbe900d2bb0 0x7fbe90151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 --2- 172.21.15.1:0/2459849791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe900d1000 0x7fbe901515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 >> 172.21.15.1:0/2459849791 conn(0x7fbe900d3ec0 msgr2=0x7fbe900d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 shutdown_connections 2024-03-24T10:34:43.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.637+0000 7fbe9684d640 1 -- 172.21.15.1:0/2459849791 wait complete. 2024-03-24T10:34:43.639 INFO:tasks.ceph.mds.b.smithi003.stderr: -3080> 2024-03-24T10:33:59.096+0000 7f5f1fcd8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:34:43.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T10:34:43.667 INFO:tasks.ceph.mds.b.smithi003.stdout:starting mds.b at 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.752+0000 7ff1b478a640 1 Processor -- start 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.752+0000 7ff1b478a640 1 -- start start 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.752+0000 7ff1b478a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1ac15a4d0 0x7ff1ac06c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b478a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac06d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b478a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1ac06d7a0 0x7ff1ac15e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b478a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ac0cc270 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b478a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ac0b4230 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b478a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ac06ce70 con 0x7ff1ac0b4ca0 2024-03-24T10:34:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1ac15a4d0 0x7ff1ac06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1ac15a4d0 0x7ff1ac06c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52684/0 (socket says 172.21.15.1:52684) 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 -- 172.21.15.1:0/859120666 learned_addr learned my addr 172.21.15.1:0/859120666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2d27640 1 --2- 172.21.15.1:0/859120666 >> v2:172.21.15.3:3300/0 conn(0x7ff1ac06d7a0 0x7ff1ac15e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 -- 172.21.15.1:0/859120666 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 msgr2=0x7ff1ac06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac06d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 -- 172.21.15.1:0/859120666 >> v2:172.21.15.3:3300/0 conn(0x7ff1ac06d7a0 msgr2=0x7ff1ac15e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 --2- 172.21.15.1:0/859120666 >> v2:172.21.15.3:3300/0 conn(0x7ff1ac06d7a0 0x7ff1ac15e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b2526640 1 -- 172.21.15.1:0/859120666 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac06dc00 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.753+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b2526640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac06c930 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac06ed30 tx=0x7ff1a802cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9687c0f4fafd8bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b1524640 1 -- 172.21.15.1:0/859120666 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a802dba0 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b1524640 1 -- 172.21.15.1:0/859120666 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a802dd00 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b1524640 1 -- 172.21.15.1:0/859120666 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8035660 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 -- 172.21.15.1:0/859120666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac15a4d0 msgr2=0x7ff1ac06c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac06c930 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac06ed30 tx=0x7ff1a802cdf0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 -- 172.21.15.1:0/859120666 shutdown_connections 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 --2- 172.21.15.1:0/859120666 >> v2:172.21.15.3:3300/0 conn(0x7ff1ac06d7a0 0x7ff1ac15e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 --2- 172.21.15.1:0/859120666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac06c930 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.754+0000 7ff1b478a640 1 -- 172.21.15.1:0/859120666 >> 172.21.15.1:0/859120666 conn(0x7ff1ac06b4f0 msgr2=0x7ff1ac06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.755+0000 7ff1b478a640 1 -- 172.21.15.1:0/859120666 shutdown_connections 2024-03-24T10:34:43.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.755+0000 7ff1b478a640 1 -- 172.21.15.1:0/859120666 wait complete. 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 Processor -- start 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 -- start start 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac14f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac15abf0 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac158e80 con 0x7ff1ac15a4d0 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b478a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ac159160 con 0x7ff1ac0b4ca0 2024-03-24T10:34:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.756+0000 7ff1b2526640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52688/0 (socket says 172.21.15.1:52688) 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 -- 172.21.15.1:0/1797474694 learned_addr learned my addr 172.21.15.1:0/1797474694 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2d27640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac14f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 -- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 msgr2=0x7ff1ac14f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac14f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 -- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 msgr2=0x7ff1ac1562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac1562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 -- 172.21.15.1:0/1797474694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a800a670 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2d27640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac1562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac14f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b2526640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff1a802ce30 tx=0x7ff1a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a803f600 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a803f760 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a803fb10 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac159440 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.757+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac14fc40 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.758+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac1592a0 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.758+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1a803fc70 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.759+0000 7ff1a2ffd640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 0x7ff19006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.759+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 0x7ff19006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.759+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5977+0+0 (secure 0 0 0) 0x7ff1a80ac830 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.759+0000 7ff1b1d25640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 0x7ff19006dd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cdc10 tx=0x7ff19c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:43.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.763+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff1ac1592a0 con 0x7ff1ac06d7a0 2024-03-24T10:34:43.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:43.981+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff1ac1592a0 con 0x7ff1ac06d7a0 2024-03-24T10:34:44.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.634+0000 7ff1a2ffd640 1 -- 172.21.15.1:0/1797474694 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7ff1ac1592a0 con 0x7ff1ac06d7a0 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 msgr2=0x7ff19006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 0x7ff19006dd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cdc10 tx=0x7ff19c008040 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 msgr2=0x7ff1ac1588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff1a802ce30 tx=0x7ff1a80046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 shutdown_connections 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff19006b840 0x7ff19006dd00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1ac15a4d0 0x7ff1ac1562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff1ac0b4ca0 0x7ff1ac14f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 --2- 172.21.15.1:0/1797474694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1ac06d7a0 0x7ff1ac1588a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 >> 172.21.15.1:0/1797474694 conn(0x7ff1ac06b4f0 msgr2=0x7ff1ac151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 shutdown_connections 2024-03-24T10:34:44.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.638+0000 7ff1b478a640 1 -- 172.21.15.1:0/1797474694 wait complete. 2024-03-24T10:34:44.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T10:34:44.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.764+0000 7f75539b6640 1 Processor -- start 2024-03-24T10:34:44.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.764+0000 7f75539b6640 1 -- start start 2024-03-24T10:34:44.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f75539b6640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f754c0d2a20 0x7f754c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f75539b6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754c0b0b70 con 0x7f754c0b2310 2024-03-24T10:34:44.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f75539b6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754c0d1d60 con 0x7f754c15e390 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f75539b6640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f754c0b0de0 con 0x7f754c0d2a20 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7551f53640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f754c0b2310 0x7f754c0cf010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46780/0 (socket says 172.21.15.1:46780) 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7551f53640 1 -- 172.21.15.1:0/2101919200 learned_addr learned my addr 172.21.15.1:0/2101919200 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7550f51640 1 --2- 172.21.15.1:0/2101919200 >> v2:172.21.15.3:3300/0 conn(0x7f754c0d2a20 0x7f754c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608125307 0 0) 0x7f754c0b0b70 con 0x7f754c0b2310 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7538003520 con 0x7f754c0b2310 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618322909 0 0) 0x7f754c0d1d60 con 0x7f754c15e390 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f754c0b0b70 con 0x7f754c15e390 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7550f51640 1 -- 172.21.15.1:0/2101919200 >> v1:172.21.15.3:6789/0 conn(0x7f754c15e390 legacy=0x7f754c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7550f51640 1 -- 172.21.15.1:0/2101919200 >> v1:172.21.15.1:6789/0 conn(0x7f754c0b2310 legacy=0x7f754c0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.765+0000 7f7550f51640 1 -- 172.21.15.1:0/2101919200 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c15e770 con 0x7f754c0d2a20 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.766+0000 7f7550f51640 1 --2- 172.21.15.1:0/2101919200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c0d2a20 0x7f754c0cead0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f753c0097a0 tx=0x7f753c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29c2b7ee3668a2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c030610 con 0x7f754c0d2a20 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f753c030770 con 0x7f754c0d2a20 2024-03-24T10:34:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f7543fff640 1 -- 172.21.15.1:0/2101919200 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c030ae0 con 0x7f754c0d2a20 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 -- 172.21.15.1:0/2101919200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c0d2a20 msgr2=0x7f754c0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 --2- 172.21.15.1:0/2101919200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c0d2a20 0x7f754c0cead0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f753c0097a0 tx=0x7f753c02b960 comp rx=0 tx=0).stop 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 -- 172.21.15.1:0/2101919200 shutdown_connections 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 --2- 172.21.15.1:0/2101919200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c0d2a20 0x7f754c0cead0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 -- 172.21.15.1:0/2101919200 >> 172.21.15.1:0/2101919200 conn(0x7f754c0d3ee0 msgr2=0x7f754c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 -- 172.21.15.1:0/2101919200 shutdown_connections 2024-03-24T10:34:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.767+0000 7f75539b6640 1 -- 172.21.15.1:0/2101919200 wait complete. 2024-03-24T10:34:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.768+0000 7f75539b6640 1 Processor -- start 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 -- start start 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 0x7f754c0b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 0x7f754c0b9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f754c0b8370 con 0x7f754c0b2310 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f754c0b74c0 con 0x7f754c15e390 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f75539b6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f754c0b76f0 con 0x7f754c0d2a20 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f7551752640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f7551752640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52692/0 (socket says 172.21.15.1:52692) 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.769+0000 7f7551752640 1 -- 172.21.15.1:0/117444228 learned_addr learned my addr 172.21.15.1:0/117444228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7550f51640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 0x7f754c0b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551f53640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 0x7f754c0b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 -- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 msgr2=0x7f754c0b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 0x7f754c0b8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 -- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 msgr2=0x7f754c0b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 0x7f754c0b9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 -- 172.21.15.1:0/117444228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c009400 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551f53640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 0x7f754c0b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7550f51640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 0x7f754c0b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7551752640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f754800c430 tx=0x7f754800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75480052d0 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7548005450 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c0b79d0 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548003830 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.770+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c0b7f90 con 0x7f754c0b2310 2024-03-24T10:34:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.771+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754c0b84b0 con 0x7f754c0b2310 2024-03-24T10:34:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.772+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7548003990 con 0x7f754c0b2310 2024-03-24T10:34:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.772+0000 7f7541ffb640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 0x7f753006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.773+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5977+0+0 (secure 0 0 0) 0x7f7548089890 con 0x7f754c0b2310 2024-03-24T10:34:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.773+0000 7f7550f51640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 0x7f753006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.773+0000 7f7550f51640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 0x7f753006dcd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f753c036000 tx=0x7f753c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:44.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.777+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f754c0b84b0 con 0x7f754c0b2310 2024-03-24T10:34:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:44.961+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f754c0b84b0 con 0x7f754c0b2310 2024-03-24T10:34:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.636+0000 7f7541ffb640 1 -- 172.21.15.1:0/117444228 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7f7548057d60 con 0x7f754c0b2310 2024-03-24T10:34:45.636 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-24T10:34:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.639+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 msgr2=0x7f753006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.639+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 0x7f753006dcd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f753c036000 tx=0x7f753c03b040 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.639+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 msgr2=0x7f754c153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.639+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f754800c430 tx=0x7f754800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 shutdown_connections 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f753006b810 0x7f753006dcd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f754c15e390 0x7f754c0b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f754c0d2a20 0x7f754c0b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 --2- 172.21.15.1:0/117444228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f754c0b2310 0x7f754c153220 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 >> 172.21.15.1:0/117444228 conn(0x7f754c0d3ee0 msgr2=0x7f754c160780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 shutdown_connections 2024-03-24T10:34:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.641+0000 7f75539b6640 1 -- 172.21.15.1:0/117444228 wait complete. 2024-03-24T10:34:45.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T10:34:45.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 Processor -- start 2024-03-24T10:34:45.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 -- start start 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 0x7fe8240d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe824162610 0x7fe8240d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8240c7a40 con 0x7fe8240b4e40 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe8240cfd50 con 0x7fe824162610 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.763+0000 7fe82b09d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe8240b55b0 con 0x7fe8241615f0 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe824162610 0x7fe8240d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe824162610 0x7fe8240d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52706/0 (socket says 172.21.15.1:52706) 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 -- 172.21.15.1:0/2586608183 learned_addr learned my addr 172.21.15.1:0/2586608183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe828e39640 1 --2- 172.21.15.1:0/2586608183 >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 0x7fe8240d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 -- 172.21.15.1:0/2586608183 >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 msgr2=0x7fe8240d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 --2- 172.21.15.1:0/2586608183 >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 0x7fe8240d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 -- 172.21.15.1:0/2586608183 >> v1:172.21.15.1:6789/0 conn(0x7fe8240b4e40 legacy=0x7fe824161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe82963a640 1 -- 172.21.15.1:0/2586608183 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8240b52d0 con 0x7fe824162610 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe8237fe640 1 -- 172.21.15.1:0/2586608183 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331946912 0 0) 0x7fe8240c7a40 con 0x7fe8240b4e40 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.764+0000 7fe828e39640 1 --2- 172.21.15.1:0/2586608183 >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 0x7fe8240d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82963a640 1 --2- 172.21.15.1:0/2586608183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8240d1c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b69f0 tx=0x7fe81c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30965b97fc55f713 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe8237fe640 1 -- 172.21.15.1:0/2586608183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c02cb20 con 0x7fe824162610 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe8237fe640 1 -- 172.21.15.1:0/2586608183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe81c02cc80 con 0x7fe824162610 2024-03-24T10:34:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe8237fe640 1 -- 172.21.15.1:0/2586608183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0343f0 con 0x7fe824162610 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 -- 172.21.15.1:0/2586608183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 msgr2=0x7fe8240d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2586608183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8240d1c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b69f0 tx=0x7fe81c02bf80 comp rx=0 tx=0).stop 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 -- 172.21.15.1:0/2586608183 shutdown_connections 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2586608183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8240d1c70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2586608183 >> v2:172.21.15.3:3300/0 conn(0x7fe8241615f0 0x7fe8240d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.765+0000 7fe82b09d640 1 -- 172.21.15.1:0/2586608183 >> 172.21.15.1:0/2586608183 conn(0x7fe8240d3ef0 msgr2=0x7fe8240d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.766+0000 7fe82b09d640 1 -- 172.21.15.1:0/2586608183 shutdown_connections 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.766+0000 7fe82b09d640 1 -- 172.21.15.1:0/2586608183 wait complete. 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 Processor -- start 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 -- start start 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8241615f0 0x7fe824159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8240b74a0 0x7fe8241f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe824160820 con 0x7fe824162610 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe824152e70 con 0x7fe8241615f0 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.767+0000 7fe82b09d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe8241f7de0 con 0x7fe8240b74a0 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52712/0 (socket says 172.21.15.1:52712) 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 -- 172.21.15.1:0/2092492043 learned_addr learned my addr 172.21.15.1:0/2092492043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe828e39640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8241615f0 0x7fe824159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe82963a640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8240b74a0 0x7fe8241f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 -- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8240b74a0 msgr2=0x7fe8241f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8240b74a0 0x7fe8241f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 -- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8241615f0 msgr2=0x7fe824159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8241615f0 0x7fe824159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 -- 172.21.15.1:0/2092492043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c009b30 con 0x7fe824162610 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe823fff640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe81000c520 tx=0x7fe81000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8100035f0 con 0x7fe824162610 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe810003770 con 0x7fe824162610 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8240b79a0 con 0x7fe824162610 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810003b20 con 0x7fe824162610 2024-03-24T10:34:45.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.768+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe824160440 con 0x7fe824162610 2024-03-24T10:34:45.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.769+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe824160960 con 0x7fe824162610 2024-03-24T10:34:45.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.770+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe810003c80 con 0x7fe824162610 2024-03-24T10:34:45.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.770+0000 7fe8217fa640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 0x7fe80806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:45.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.771+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5539+0+0 (secure 0 0 0) 0x7fe810089640 con 0x7fe824162610 2024-03-24T10:34:45.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.771+0000 7fe828e39640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 0x7fe80806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:45.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.771+0000 7fe828e39640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 0x7fe80806dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b9d00 tx=0x7fe818008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:45.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.775+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe824160960 con 0x7fe824162610 2024-03-24T10:34:45.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:45.958+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe824160960 con 0x7fe824162610 2024-03-24T10:34:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.644+0000 7fe8217fa640 1 -- 172.21.15.1:0/2092492043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7fe824160960 con 0x7fe824162610 2024-03-24T10:34:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-24T10:34:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.647+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 msgr2=0x7fe80806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.647+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 0x7fe80806dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b9d00 tx=0x7fe818008040 comp rx=0 tx=0).stop 2024-03-24T10:34:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.647+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 msgr2=0x7fe8241f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.647+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe81000c520 tx=0x7fe81000c9e0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 shutdown_connections 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe80806b810 0x7fe80806dcd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8240b74a0 0x7fe8241f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe824162610 0x7fe8241f3220 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 --2- 172.21.15.1:0/2092492043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8241615f0 0x7fe824159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.648+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 >> 172.21.15.1:0/2092492043 conn(0x7fe8240d3ef0 msgr2=0x7fe8240c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.649+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 shutdown_connections 2024-03-24T10:34:46.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.649+0000 7fe82b09d640 1 -- 172.21.15.1:0/2092492043 wait complete. 2024-03-24T10:34:46.667 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T10:34:46.667 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T10:34:46.667 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T10:34:46.667 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:46.667 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T10:34:46.667 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T10:34:46.668 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T10:34:46.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:46.668 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T10:34:46.668 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T10:34:46.668 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T10:34:46.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:46.668 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T10:34:46.669 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T10:34:46.669 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T10:34:46.669 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:34:46.678 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T10:34:46.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T10:34:46.679 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T10:34:46.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T10:34:46.683 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T10:34:46.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T10:34:46.686 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T10:34:46.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T10:34:46.721 INFO:tasks.ceph.mds.a:Started 2024-03-24T10:34:46.721 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:46.722 INFO:tasks.ceph.mds.c:Started 2024-03-24T10:34:46.722 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:46.726 INFO:tasks.ceph.mds.b:Started 2024-03-24T10:34:46.726 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:46.727 INFO:tasks.ceph.mds.d:Started 2024-03-24T10:34:46.727 DEBUG:teuthology.parallel:result is None 2024-03-24T10:34:46.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T10:34:46.806 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-24T10:34:46.806 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-24T10:34:46.816 INFO:tasks.ceph.mds.b.smithi003.stdout:starting mds.b at 2024-03-24T10:34:46.817 INFO:tasks.ceph.mds.d.smithi003.stdout:starting mds.d at 2024-03-24T10:34:46.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.845+0000 7f95e5c3b640 1 Processor -- start 2024-03-24T10:34:46.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.845+0000 7f95e5c3b640 1 -- start start 2024-03-24T10:34:46.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95e5c3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f95e00ceac0 0x7f95e00b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95e5c3b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e00d2e10 con 0x7f95e015e150 2024-03-24T10:34:46.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95e5c3b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e00ceed0 con 0x7f95e015e530 2024-03-24T10:34:46.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95e5c3b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f95e00cf1a0 con 0x7f95e00ceac0 2024-03-24T10:34:46.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95dffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f95e00ceac0 0x7f95e00b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95df7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f95e015e150 0x7f95e00b4ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46818/0 (socket says 172.21.15.1:46818) 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95df7fe640 1 -- 172.21.15.1:0/2198121726 learned_addr learned my addr 172.21.15.1:0/2198121726 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95dffff640 1 -- 172.21.15.1:0/2198121726 >> v1:172.21.15.3:6789/0 conn(0x7f95e015e530 legacy=0x7f95e00b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95dffff640 1 -- 172.21.15.1:0/2198121726 >> v1:172.21.15.1:6789/0 conn(0x7f95e015e150 legacy=0x7f95e00b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95de7fc640 1 -- 172.21.15.1:0/2198121726 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037437667 0 0) 0x7f95e00ceed0 con 0x7f95e015e530 2024-03-24T10:34:46.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.846+0000 7f95dffff640 1 -- 172.21.15.1:0/2198121726 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e00cf480 con 0x7f95e00ceac0 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95dffff640 1 --2- 172.21.15.1:0/2198121726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e00ceac0 0x7f95e00b6f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950b9ece87a1f79e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95de7fc640 1 -- 172.21.15.1:0/2198121726 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d802cab0 con 0x7f95e00ceac0 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95de7fc640 1 -- 172.21.15.1:0/2198121726 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d802cc10 con 0x7f95e00ceac0 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95de7fc640 1 -- 172.21.15.1:0/2198121726 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8034440 con 0x7f95e00ceac0 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95e5c3b640 1 -- 172.21.15.1:0/2198121726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e00ceac0 msgr2=0x7f95e00b6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/2198121726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e00ceac0 0x7f95e00b6f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d802bd80 comp rx=0 tx=0).stop 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.847+0000 7f95e5c3b640 1 -- 172.21.15.1:0/2198121726 shutdown_connections 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.848+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/2198121726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e00ceac0 0x7f95e00b6f40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.848+0000 7f95e5c3b640 1 -- 172.21.15.1:0/2198121726 >> 172.21.15.1:0/2198121726 conn(0x7f95e009d1b0 msgr2=0x7f95e00d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.848+0000 7f95e5c3b640 1 -- 172.21.15.1:0/2198121726 shutdown_connections 2024-03-24T10:34:46.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.848+0000 7f95e5c3b640 1 -- 172.21.15.1:0/2198121726 wait complete. 2024-03-24T10:34:46.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.849+0000 7f95e5c3b640 1 Processor -- start 2024-03-24T10:34:46.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.849+0000 7f95e5c3b640 1 -- start start 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 0x7f95e00a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e015e150 0x7f95e01f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f95e00a0800 con 0x7f95e015e530 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f95e00a0ae0 con 0x7f95e015e150 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f95e00a0dc0 con 0x7f95e00ceac0 2024-03-24T10:34:46.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52778/0 (socket says 172.21.15.1:52778) 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 -- 172.21.15.1:0/3193140471 learned_addr learned my addr 172.21.15.1:0/3193140471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95df7fe640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 0x7f95e00a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95deffd640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e015e150 0x7f95e01f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 -- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 msgr2=0x7f95e00a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 0x7f95e00a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 -- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e015e150 msgr2=0x7f95e01f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e015e150 0x7f95e01f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 -- 172.21.15.1:0/3193140471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d8009a70 con 0x7f95e015e530 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95df7fe640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 0x7f95e00a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95dffff640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d803e610 con 0x7f95e015e530 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d803e790 con 0x7f95e015e530 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e00a10a0 con 0x7f95e015e530 2024-03-24T10:34:46.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d803eb40 con 0x7f95e015e530 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.850+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e01f4b40 con 0x7f95e015e530 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.851+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e01f4cd0 con 0x7f95e015e530 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.852+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f95d803b050 con 0x7f95e015e530 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.852+0000 7f95d7fff640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 0x7f95c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.852+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4943+0+0 (secure 0 0 0) 0x7f95d80aca10 con 0x7f95e015e530 2024-03-24T10:34:46.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.852+0000 7f95df7fe640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 0x7f95c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:46.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.853+0000 7f95df7fe640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 0x7f95c006dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95d0000f60 tx=0x7f95d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:46.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:46.856+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f95d8077690 con 0x7f95e015e530 2024-03-24T10:34:47.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.050+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f95e004f4e0 con 0x7f95e015e530 2024-03-24T10:34:47.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.656+0000 7f95d7fff640 1 -- 172.21.15.1:0/3193140471 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) ==== 97+0+0 (secure 0 0 0) 0x7f95d804e080 con 0x7f95e015e530 2024-03-24T10:34:47.656 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-24T10:34:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 msgr2=0x7f95c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 0x7f95c006dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95d0000f60 tx=0x7f95d0009040 comp rx=0 tx=0).stop 2024-03-24T10:34:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 msgr2=0x7f95e00c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d8004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 shutdown_connections 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f95c006b810 0x7f95c006dcd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e015e530 0x7f95e00c9eb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95e015e150 0x7f95e01f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 --2- 172.21.15.1:0/3193140471 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f95e00ceac0 0x7f95e00a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.659+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 >> 172.21.15.1:0/3193140471 conn(0x7f95e009d1b0 msgr2=0x7f95e00d3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.660+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 shutdown_connections 2024-03-24T10:34:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.660+0000 7f95e5c3b640 1 -- 172.21.15.1:0/3193140471 wait complete. 2024-03-24T10:34:47.675 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.782+0000 7f841c855640 1 Processor -- start 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.782+0000 7f841c855640 1 -- start start 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841c855640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841c855640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84180d1000 0x7f84180b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841c855640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841815e770 con 0x7f841815e390 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841c855640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f84180b3190 con 0x7f84180d1000 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841c855640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f84180d4e40 con 0x7f84180d2bb0 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f8415d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f8416d9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84180d1000 0x7f84180b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f8415d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:42742/0 (socket says 172.21.15.1:42742) 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f841659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f841815e390 0x7f84180d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:59012/0 (socket says 172.21.15.1:59012) 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f8416d9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84180d1000 0x7f84180b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59828/0 (socket says 172.21.15.1:59828) 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.783+0000 7f8415d9b640 1 -- 172.21.15.1:0/3234861352 learned_addr learned my addr 172.21.15.1:0/3234861352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.784+0000 7f8416d9d640 1 -- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 msgr2=0x7f84180b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.784+0000 7f8416d9d640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.784+0000 7f8416d9d640 1 -- 172.21.15.1:0/3234861352 >> v1:172.21.15.3:6789/0 conn(0x7f841815e390 legacy=0x7f84180d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.784+0000 7f8416d9d640 1 -- 172.21.15.1:0/3234861352 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84180d1e40 con 0x7f84180d1000 2024-03-24T10:34:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.784+0000 7f8415d9b640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f8416d9d640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b40d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8408009d90 tx=0x7f840802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73840588f3007356 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841559a640 1 -- 172.21.15.1:0/3234861352 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840802cb30 con 0x7f84180d1000 2024-03-24T10:34:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841559a640 1 -- 172.21.15.1:0/3234861352 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840802cc90 con 0x7f84180d1000 2024-03-24T10:34:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841559a640 1 -- 172.21.15.1:0/3234861352 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408034440 con 0x7f84180d1000 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 -- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 msgr2=0x7f84180b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b40d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8408009d90 tx=0x7f840802bd80 comp rx=0 tx=0).stop 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 -- 172.21.15.1:0/3234861352 shutdown_connections 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b40d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 --2- 172.21.15.1:0/3234861352 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84180b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 -- 172.21.15.1:0/3234861352 >> 172.21.15.1:0/3234861352 conn(0x7f84180d3ec0 msgr2=0x7f84180d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 -- 172.21.15.1:0/3234861352 shutdown_connections 2024-03-24T10:34:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.785+0000 7f841c855640 1 -- 172.21.15.1:0/3234861352 wait complete. 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 Processor -- start 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 -- start start 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84181f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f84180b9520 con 0x7f841815e390 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f841814b470 con 0x7f84180d1000 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.787+0000 7f841c855640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f84180b8b80 con 0x7f84180d2bb0 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52794/0 (socket says 172.21.15.1:52794) 2024-03-24T10:34:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 -- 172.21.15.1:0/1241605410 learned_addr learned my addr 172.21.15.1:0/1241605410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f841659c640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8415d9b640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84181f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 -- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 msgr2=0x7f84181f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84181f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 -- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 msgr2=0x7f84180b9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 -- 172.21.15.1:0/1241605410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000094d0 con 0x7f841815e390 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f841659c640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f8416d9d640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8408009d90 tx=0x7f8408004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408040450 con 0x7f841815e390 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84080405d0 con 0x7f841815e390 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8408009a70 con 0x7f841815e390 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408040980 con 0x7f841815e390 2024-03-24T10:34:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.788+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84180b9140 con 0x7f841815e390 2024-03-24T10:34:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.789+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84180b9660 con 0x7f841815e390 2024-03-24T10:34:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.790+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8408040ae0 con 0x7f841815e390 2024-03-24T10:34:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.790+0000 7f840effd640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 0x7f83f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.790+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f84080ac930 con 0x7f841815e390 2024-03-24T10:34:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.790+0000 7f841659c640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 0x7f83f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.791+0000 7f841659c640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 0x7f83f806dd00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8410007fd0 tx=0x7f8410007d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:47.794+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f84180b9660 con 0x7f841815e390 2024-03-24T10:34:48.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.042+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f84180b9660 con 0x7f841815e390 2024-03-24T10:34:48.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.043+0000 7f840effd640 1 -- 172.21.15.1:0/1241605410 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f84180b9660 con 0x7f841815e390 2024-03-24T10:34:48.043 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:48.043 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmJeY4FBAA68H0WQ/mqjsr0lZfBIvuIg==", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm2p+QIBAAokf+P5vhhBqszwKd+fn11w==", 2024-03-24T10:34:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmjiKEGxAA9ZjhlLO78jdVtdk4FZILhA==", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm+dgUKBAAG62jv3f5tbgZc3yHG/BiFQ==", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBcAQBmcF+vEhAA6d6m64/OD24zfbP1RjYR1w==", 2024-03-24T10:34:48.046 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBfAQBmOi2fERAAJ3x4Y4d96THYK1bBkTqqPg==", 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBiAQBmhsQJEBAAGwQdvTTDG3liCqR+8Y1++w==", 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBlAQBmzW42DhAAopMY+PpT0hSNL1Y/h0rOmQ==", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBrAQBmqZxLFRAAoXQjHMDUYKqwpqKunxNuzg==", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBuAQBmIQoSFBAAgw2p6P1rqsdn94zL8BCmDg==", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBxAQBmnxrVEhAAthwnDMkD5AyCr0gCPvE3ZQ==", 2024-03-24T10:34:48.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB0AQBm8UNjERAARFrNk44p63MiF0L0YVsAqw==", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw==", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmO0d+NRAAhw9H+eNh7wKil/BAbqbQSQ==", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:34:48.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm1Z8YMhAArYu32NrH0OcQYcX13UCsnQ==", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-24T10:34:48.054 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm9xNHOxAAAxXBCXt6EiKWgso4flywRQ==", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBZAQBmpZqfMhAAB5KUBr9dFz54a442ZCMWIg==", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mds", 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBm1fK2DBAAmatKLklTaICeot4gPGMqUQ==", 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mds" 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mgr", 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBmofy2DBAAWVpd5NZgNpEwVlZccdZ7iQ==", 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-osd", 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBm7AO3DBAA3IEh+pBscRt844fInvwZXg==", 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-osd" 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd", 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBmJQu3DBAApCTvLnxXJ4Ri4z3C/5+5Vw==", 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-24T10:34:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBmShK3DBAAePif/C8B4cmd95azxnST5Q==", 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rgw", 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB6AQBmXBm3DBAA+bTct0e4OHkQR2tb0/LrAA==", 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmBLzREBAA9Hph9Xljf0eGaV92TSLlfg==", 2024-03-24T10:34:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmHPHHDBAAIuuLFGLFsRogrQjUx44H7A==", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:48.059 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-24T10:34:48.060 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:34:48.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 msgr2=0x7f83f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 0x7f83f806dd00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8410007fd0 tx=0x7f8410007d60 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 msgr2=0x7f84180cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8408009d90 tx=0x7f8408004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 shutdown_connections 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f83f806b840 0x7f83f806dd00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f841815e390 0x7f84180cd4b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f84180d2bb0 0x7f84181f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 --2- 172.21.15.1:0/1241605410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84180d1000 0x7f84180b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 >> 172.21.15.1:0/1241605410 conn(0x7f84180d3ec0 msgr2=0x7f84180b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 shutdown_connections 2024-03-24T10:34:48.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.045+0000 7f841c855640 1 -- 172.21.15.1:0/1241605410 wait complete. 2024-03-24T10:34:48.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-24T10:34:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 Processor -- start 2024-03-24T10:34:48.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 -- start start 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0d27a0 0x7f87ec0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87ec0cc790 con 0x7f87ec0cd830 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f87ec0d1930 con 0x7f87ec0cfd00 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87f13af640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f87ec0d0be0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.165+0000 7f87eaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52798/0 (socket says 172.21.15.1:52798) 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 -- 172.21.15.1:0/1865438706 learned_addr learned my addr 172.21.15.1:0/1865438706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eb7fe640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0d27a0 0x7f87ec0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 -- 172.21.15.1:0/1865438706 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0d27a0 msgr2=0x7f87ec0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0d27a0 0x7f87ec0b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 -- 172.21.15.1:0/1865438706 >> v1:172.21.15.3:6789/0 conn(0x7f87ec0cd830 legacy=0x7f87ec0d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.166+0000 7f87eaffd640 1 -- 172.21.15.1:0/1865438706 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ec0cca70 con 0x7f87ec0cfd00 2024-03-24T10:34:48.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87eaffd640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009870 tx=0x7f87dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91ca04f685274ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87e9ffb640 1 -- 172.21.15.1:0/1865438706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc030840 con 0x7f87ec0cfd00 2024-03-24T10:34:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87e9ffb640 1 -- 172.21.15.1:0/1865438706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87dc0309a0 con 0x7f87ec0cfd00 2024-03-24T10:34:48.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87e9ffb640 1 -- 172.21.15.1:0/1865438706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc030d30 con 0x7f87ec0cfd00 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 -- 172.21.15.1:0/1865438706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0cfd00 msgr2=0x7f87ec0cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009870 tx=0x7f87dc02b980 comp rx=0 tx=0).stop 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 -- 172.21.15.1:0/1865438706 shutdown_connections 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0d27a0 0x7f87ec0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 --2- 172.21.15.1:0/1865438706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0cfd00 0x7f87ec0cf0a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 -- 172.21.15.1:0/1865438706 >> 172.21.15.1:0/1865438706 conn(0x7f87ec0d1c80 msgr2=0x7f87ec0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 -- 172.21.15.1:0/1865438706 shutdown_connections 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.167+0000 7f87f13af640 1 -- 172.21.15.1:0/1865438706 wait complete. 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 Processor -- start 2024-03-24T10:34:48.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 -- start start 2024-03-24T10:34:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87ec0cd830 0x7f87ec151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0cfd00 0x7f87ec0af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec0b1da0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec0b1470 con 0x7f87ec0cd830 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.169+0000 7f87f13af640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f87ec0b16e0 con 0x7f87ec0cfd00 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52806/0 (socket says 172.21.15.1:52806) 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 -- 172.21.15.1:0/4221616090 learned_addr learned my addr 172.21.15.1:0/4221616090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eaffd640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87ec0cd830 0x7f87ec151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87ea7fc640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0cfd00 0x7f87ec0af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 -- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0cfd00 msgr2=0x7f87ec0af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0cfd00 0x7f87ec0af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 -- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87ec0cd830 msgr2=0x7f87ec151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87ec0cd830 0x7f87ec151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 -- 172.21.15.1:0/4221616090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc0094d0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87eb7fe640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f87e4007e70 tx=0x7f87e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4014620 con 0x7f87ec0d27a0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e40147a0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ec0b19c0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4014b50 con 0x7f87ec0d27a0 2024-03-24T10:34:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.170+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ec1524a0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.171+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ec04f4e0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.172+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f87e4014cb0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.172+0000 7f87e37fe640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 0x7f87cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.172+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f87e408a7c0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.172+0000 7f87eaffd640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 0x7f87cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.173+0000 7f87eaffd640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 0x7f87cc06dcd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87dc036000 tx=0x7f87dc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.176+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f87e40555c0 con 0x7f87ec0d27a0 2024-03-24T10:34:48.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.426+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f87ec0ce470 con 0x7f87ec0d27a0 2024-03-24T10:34:48.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.428+0000 7f87e37fe640 1 -- 172.21.15.1:0/4221616090 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f87e4058c80 con 0x7f87ec0d27a0 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 msgr2=0x7f87cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 0x7f87cc06dcd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87dc036000 tx=0x7f87dc008030 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 msgr2=0x7f87ec0b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f87e4007e70 tx=0x7f87e400e4b0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 shutdown_connections 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f87cc06b810 0x7f87cc06dcd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec0d27a0 0x7f87ec0b21c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f87ec0cfd00 0x7f87ec0af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 --2- 172.21.15.1:0/4221616090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87ec0cd830 0x7f87ec151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 >> 172.21.15.1:0/4221616090 conn(0x7f87ec0d1c80 msgr2=0x7f87ec0d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 shutdown_connections 2024-03-24T10:34:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.430+0000 7f87f13af640 1 -- 172.21.15.1:0/4221616090 wait complete. 2024-03-24T10:34:48.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-24T10:34:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.545+0000 7f608c076640 1 Processor -- start 2024-03-24T10:34:48.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.545+0000 7f608c076640 1 -- start start 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f608c076640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6084072710 0x7f60840b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f608c076640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6084072c10 con 0x7f608406b7b0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f608c076640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608406dde0 con 0x7f608415e0a0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f608c076640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f608406e0c0 con 0x7f6084072710 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6089611640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f608406b7b0 0x7f60840b4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46830/0 (socket says 172.21.15.1:46830) 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6089611640 1 -- 172.21.15.1:0/2776746157 learned_addr learned my addr 172.21.15.1:0/2776746157 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729851614 0 0) 0x7f6084072c10 con 0x7f608406b7b0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6070003520 con 0x7f608406b7b0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f608a613640 1 --2- 172.21.15.1:0/2776746157 >> v2:172.21.15.3:3300/0 conn(0x7f6084072710 0x7f60840b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1438699911 0 0) 0x7f6070003520 con 0x7f608406b7b0 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 >> v2:172.21.15.3:3300/0 conn(0x7f6084072710 msgr2=0x7f60840b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 --2- 172.21.15.1:0/2776746157 >> v2:172.21.15.3:3300/0 conn(0x7f6084072710 0x7f60840b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 >> v1:172.21.15.3:6789/0 conn(0x7f608415e0a0 legacy=0x7f608406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608406e3a0 con 0x7f608406b7b0 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f6074001670 con 0x7f608406b7b0 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 >> v1:172.21.15.1:6789/0 conn(0x7f608406b7b0 legacy=0x7f60840b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070003970 0x7f6070003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60700047c0 0x7f6070004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 0x7f607000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f608406e0c0 con 0x7f6070005180 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f607000c020 con 0x7f6070003970 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.546+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f607000c1e0 con 0x7f60700047c0 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 0x7f607000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f6089e12640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070003970 0x7f6070003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f6089611640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60700047c0 0x7f6070004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 -- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60700047c0 msgr2=0x7f6070004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f60700047c0 0x7f6070004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 -- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070003970 msgr2=0x7f6070003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070003970 0x7f6070003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 -- 172.21.15.1:0/2776746157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6070003520 con 0x7f6070005180 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 -- 172.21.15.1:0/2776746157 reap_dead start 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.547+0000 7f608a613640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 0x7f607000b980 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f607c009f90 tx=0x7f60700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3fc9f9d8cd18fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c02cc20 con 0x7f6070005180 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f607c00f040 con 0x7f6070005180 2024-03-24T10:34:48.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f6088e10640 1 -- 172.21.15.1:0/2776746157 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c003990 con 0x7f6070005180 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 -- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 msgr2=0x7f607000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 0x7f607000b980 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f607c009f90 tx=0x7f60700057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 -- 172.21.15.1:0/2776746157 shutdown_connections 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 --2- 172.21.15.1:0/2776746157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6070005180 0x7f607000b980 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 -- 172.21.15.1:0/2776746157 >> 172.21.15.1:0/2776746157 conn(0x7f60840b2820 msgr2=0x7f60840b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 -- 172.21.15.1:0/2776746157 shutdown_connections 2024-03-24T10:34:48.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.548+0000 7f608c076640 1 -- 172.21.15.1:0/2776746157 wait complete. 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.549+0000 7f608c076640 1 Processor -- start 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.549+0000 7f608c076640 1 -- start start 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070005180 0x7f60841f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6084160610 0x7f60840cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6084160dd0 con 0x7f60840ce240 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f60841f49f0 con 0x7f6070005180 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608c076640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60840cddb0 con 0x7f6084160610 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608a613640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608a613640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52820/0 (socket says 172.21.15.1:52820) 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608a613640 1 -- 172.21.15.1:0/3273023130 learned_addr learned my addr 172.21.15.1:0/3273023130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f6089e12640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070005180 0x7f60841f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f6089611640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6084160610 0x7f60840cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608a613640 1 -- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6084160610 msgr2=0x7f60840cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.550+0000 7f608a613640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6084160610 0x7f60840cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608a613640 1 -- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070005180 msgr2=0x7f60841f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608a613640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070005180 0x7f60841f44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608a613640 1 -- 172.21.15.1:0/3273023130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608415d1c0 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608a613640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f607c02c690 tx=0x7f607c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c004230 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f607c004390 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6084160af0 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c004740 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.551+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60840cda40 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.552+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60840cd5c0 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.552+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f607c03f470 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.553+0000 7f607a7fc640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 0x7f606806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.553+0000 7f6089e12640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 0x7f606806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.553+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f607c0ac700 con 0x7f60840ce240 2024-03-24T10:34:48.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.553+0000 7f6089e12640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 0x7f606806dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6080000c00 tx=0x7f6080009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.557+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f60840cd5c0 con 0x7f60840ce240 2024-03-24T10:34:48.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.808+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f60840cd5c0 con 0x7f60840ce240 2024-03-24T10:34:48.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.810+0000 7f607a7fc640 1 -- 172.21.15.1:0/3273023130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f60840cd5c0 con 0x7f60840ce240 2024-03-24T10:34:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 msgr2=0x7f606806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 0x7f606806dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6080000c00 tx=0x7f6080009040 comp rx=0 tx=0).stop 2024-03-24T10:34:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 msgr2=0x7f60841553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f607c02c690 tx=0x7f607c009750 comp rx=0 tx=0).stop 2024-03-24T10:34:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 shutdown_connections 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f606806b810 0x7f606806dcd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60840ce240 0x7f60841553a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6084160610 0x7f60840cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 --2- 172.21.15.1:0/3273023130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6070005180 0x7f60841f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 >> 172.21.15.1:0/3273023130 conn(0x7f60840b2820 msgr2=0x7f60840b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 shutdown_connections 2024-03-24T10:34:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.812+0000 7f608c076640 1 -- 172.21.15.1:0/3273023130 wait complete. 2024-03-24T10:34:48.823 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-24T10:34:48.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.925+0000 7fa85fb63640 1 Processor -- start 2024-03-24T10:34:48.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.925+0000 7fa85fb63640 1 -- start start 2024-03-24T10:34:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85fb63640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa858144770 0x7fa8580d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85fb63640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8580b2aa0 0x7fa8580b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85fb63640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580cdd60 con 0x7fa858070b90 2024-03-24T10:34:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85fb63640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa85806eaa0 con 0x7fa858144770 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85fb63640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa8580b3490 con 0x7fa8580b2aa0 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d0fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa858070b90 0x7fa85806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46846/0 (socket says 172.21.15.1:46846) 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d8ff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa858144770 0x7fa8580d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d0fe640 1 -- 172.21.15.1:0/1980819205 learned_addr learned my addr 172.21.15.1:0/1980819205 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85e100640 1 --2- 172.21.15.1:0/1980819205 >> v2:172.21.15.3:3300/0 conn(0x7fa8580b2aa0 0x7fa8580b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85c8fd640 1 -- 172.21.15.1:0/1980819205 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481079634 0 0) 0x7fa8580cdd60 con 0x7fa858070b90 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85c8fd640 1 -- 172.21.15.1:0/1980819205 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa844003520 con 0x7fa858070b90 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d8ff640 1 -- 172.21.15.1:0/1980819205 >> v2:172.21.15.3:3300/0 conn(0x7fa8580b2aa0 msgr2=0x7fa8580b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d8ff640 1 --2- 172.21.15.1:0/1980819205 >> v2:172.21.15.3:3300/0 conn(0x7fa8580b2aa0 0x7fa8580b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d8ff640 1 -- 172.21.15.1:0/1980819205 >> v1:172.21.15.1:6789/0 conn(0x7fa858070b90 legacy=0x7fa85806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.926+0000 7fa85d8ff640 1 -- 172.21.15.1:0/1980819205 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85806e200 con 0x7fa858144770 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85d8ff640 1 --2- 172.21.15.1:0/1980819205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858144770 0x7fa8580d00f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa8580af6e0 tx=0x7fa85402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5076c44b08c877d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85c8fd640 1 -- 172.21.15.1:0/1980819205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854031830 con 0x7fa858144770 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85c8fd640 1 -- 172.21.15.1:0/1980819205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa854031990 con 0x7fa858144770 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85c8fd640 1 -- 172.21.15.1:0/1980819205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854031d20 con 0x7fa858144770 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85fb63640 1 -- 172.21.15.1:0/1980819205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858144770 msgr2=0x7fa8580d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.927+0000 7fa85fb63640 1 --2- 172.21.15.1:0/1980819205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858144770 0x7fa8580d00f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa8580af6e0 tx=0x7fa85402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 -- 172.21.15.1:0/1980819205 shutdown_connections 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 --2- 172.21.15.1:0/1980819205 >> v2:172.21.15.3:3300/0 conn(0x7fa8580b2aa0 0x7fa8580b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 --2- 172.21.15.1:0/1980819205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858144770 0x7fa8580d00f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 -- 172.21.15.1:0/1980819205 >> 172.21.15.1:0/1980819205 conn(0x7fa8580b17a0 msgr2=0x7fa8580b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 -- 172.21.15.1:0/1980819205 shutdown_connections 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.928+0000 7fa85fb63640 1 -- 172.21.15.1:0/1980819205 wait complete. 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.929+0000 7fa85fb63640 1 Processor -- start 2024-03-24T10:34:48.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.929+0000 7fa85fb63640 1 -- start start 2024-03-24T10:34:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580b2aa0 0x7fa8580714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 0x7fa8580719f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa858071040 con 0x7fa858070b90 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa85813b200 con 0x7fa8580b2aa0 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa85813a5f0 con 0x7fa858144770 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52834/0 (socket says 172.21.15.1:52834) 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 -- 172.21.15.1:0/2775571144 learned_addr learned my addr 172.21.15.1:0/2775571144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d0fe640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580b2aa0 0x7fa8580714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85e100640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 0x7fa8580719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 -- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 msgr2=0x7fa8580719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 0x7fa8580719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 -- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580b2aa0 msgr2=0x7fa8580714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580b2aa0 0x7fa8580714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 -- 172.21.15.1:0/2775571144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa854009400 con 0x7fa858070b90 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85e100640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 0x7fa8580719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85d8ff640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa854000c00 tx=0x7fa8540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85403a830 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa85403a9b0 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85813a8d0 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85403ad60 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.930+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85813ae90 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.931+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa85804f4e0 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.932+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa85404d450 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.932+0000 7fa84e7fc640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 0x7fa83c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.932+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fa8540abe80 con 0x7fa858070b90 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.932+0000 7fa85d0fe640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 0x7fa83c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:48.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.933+0000 7fa85d0fe640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 0x7fa83c06dcd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa8580720b0 tx=0x7fa848009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:48.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:48.937+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa854076bf0 con 0x7fa858070b90 2024-03-24T10:34:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.187+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fa85806cc90 con 0x7fa858070b90 2024-03-24T10:34:49.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.189+0000 7fa84e7fc640 1 -- 172.21.15.1:0/2775571144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7fa854035020 con 0x7fa858070b90 2024-03-24T10:34:49.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 msgr2=0x7fa83c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 0x7fa83c06dcd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa8580720b0 tx=0x7fa848009040 comp rx=0 tx=0).stop 2024-03-24T10:34:49.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 msgr2=0x7fa85815c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa854000c00 tx=0x7fa8540046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 shutdown_connections 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa83c06b810 0x7fa83c06dcd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa858144770 0x7fa8580719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580b2aa0 0x7fa8580714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 --2- 172.21.15.1:0/2775571144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa858070b90 0x7fa85815c470 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 >> 172.21.15.1:0/2775571144 conn(0x7fa8580b17a0 msgr2=0x7fa858143630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 shutdown_connections 2024-03-24T10:34:49.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.191+0000 7fa85fb63640 1 -- 172.21.15.1:0/2775571144 wait complete. 2024-03-24T10:34:49.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-24T10:34:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.304+0000 7f22cb1fd640 1 Processor -- start 2024-03-24T10:34:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.304+0000 7f22cb1fd640 1 -- start start 2024-03-24T10:34:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22cb1fd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f22c415e390 0x7f22c40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22cb1fd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c415e770 con 0x7f22c40d1000 2024-03-24T10:34:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22cb1fd640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c40d4e40 con 0x7f22c40b4ba0 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22cb1fd640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f22c40b3190 con 0x7f22c415e390 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22c3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f22c40d1000 0x7f22c40cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46862/0 (socket says 172.21.15.1:46862) 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.305+0000 7f22c3fff640 1 -- 172.21.15.1:0/3328043357 learned_addr learned my addr 172.21.15.1:0/3328043357 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836051168 0 0) 0x7f22b8001110 con 0x7f22c40d1000 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 --2- 172.21.15.1:0/3328043357 >> v2:172.21.15.3:3300/0 conn(0x7f22c415e390 0x7f22c40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22ac003520 con 0x7f22c40d1000 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3243288176 0 0) 0x7f22ac003520 con 0x7f22c40d1000 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 >> v2:172.21.15.3:3300/0 conn(0x7f22c415e390 msgr2=0x7f22c40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 --2- 172.21.15.1:0/3328043357 >> v2:172.21.15.3:3300/0 conn(0x7f22c415e390 0x7f22c40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 >> v1:172.21.15.3:6789/0 conn(0x7f22c40b4ba0 legacy=0x7f22c40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c40d1e40 con 0x7f22c40d1000 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f22b8001800 con 0x7f22c40d1000 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 >> v1:172.21.15.1:6789/0 conn(0x7f22c40d1000 legacy=0x7f22c40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 0x7f22ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22ac0047c0 0x7f22ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac0052a0 0x7f22ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac00c140 con 0x7f22ac003970 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac00c2b0 con 0x7f22ac0052a0 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f22ac00c470 con 0x7f22ac0047c0 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 0x7f22ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c3fff640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22ac0047c0 0x7f22ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c979a640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac0052a0 0x7f22ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 -- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22ac0047c0 msgr2=0x7f22ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22ac0047c0 0x7f22ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 -- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac0052a0 msgr2=0x7f22ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac0052a0 0x7f22ac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c979a640 1 -- 172.21.15.1:0/3328043357 reap_dead start 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.306+0000 7f22c8f99640 1 -- 172.21.15.1:0/3328043357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ac003520 con 0x7f22ac003970 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.307+0000 7f22c8f99640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 0x7f22ac003d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22b4009a90 tx=0x7f22ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53474027086ddc4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.307+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b402c680 con 0x7f22ac003970 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.307+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b402c7e0 con 0x7f22ac003970 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.307+0000 7f22c37fe640 1 -- 172.21.15.1:0/3328043357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b402cb70 con 0x7f22ac003970 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 -- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 msgr2=0x7f22ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 0x7f22ac003d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22b4009a90 tx=0x7f22ac0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 -- 172.21.15.1:0/3328043357 shutdown_connections 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/3328043357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22ac003970 0x7f22ac003d90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 -- 172.21.15.1:0/3328043357 >> 172.21.15.1:0/3328043357 conn(0x7f22c40d3ec0 msgr2=0x7f22c40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 -- 172.21.15.1:0/3328043357 shutdown_connections 2024-03-24T10:34:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.308+0000 7f22cb1fd640 1 -- 172.21.15.1:0/3328043357 wait complete. 2024-03-24T10:34:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.309+0000 7f22cb1fd640 1 Processor -- start 2024-03-24T10:34:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.309+0000 7f22cb1fd640 1 -- start start 2024-03-24T10:34:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 0x7f22c415eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22c4160c40 0x7f22c41610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22c40cbf70 con 0x7f22c415b030 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f22c415b9d0 con 0x7f22ac003970 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22cb1fd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f22c40cb2f0 con 0x7f22c4160c40 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52846/0 (socket says 172.21.15.1:52846) 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 -- 172.21.15.1:0/655633394 learned_addr learned my addr 172.21.15.1:0/655633394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c8f99640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 0x7f22c415eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c979a640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22c4160c40 0x7f22c41610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 -- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22c4160c40 msgr2=0x7f22c41610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22c4160c40 0x7f22c41610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 -- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 msgr2=0x7f22c415eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 0x7f22c415eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 -- 172.21.15.1:0/655633394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c40cb5d0 con 0x7f22c415b030 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c8f99640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 0x7f22c415eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.310+0000 7f22c3fff640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f22b800e760 tx=0x7f22b800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.311+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b800ee70 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.311+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b800c8d0 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.311+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c40cb8b0 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.311+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b800cc80 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.311+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c415efe0 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.312+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c40cc0b0 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.312+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22b8013420 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.312+0000 7f22c17fa640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 0x7f22b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.312+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f22b80899d0 con 0x7f22c415b030 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.313+0000 7f22c8f99640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 0x7f22b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.313+0000 7f22c8f99640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 0x7f22b006dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22b4036000 tx=0x7f22b4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.317+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22c40cc0b0 con 0x7f22c415b030 2024-03-24T10:34:49.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.567+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f22c415e770 con 0x7f22c415b030 2024-03-24T10:34:49.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.569+0000 7f22c17fa640 1 -- 172.21.15.1:0/655633394 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f22c415e770 con 0x7f22c415b030 2024-03-24T10:34:49.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 msgr2=0x7f22b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 0x7f22b006dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22b4036000 tx=0x7f22b4030000 comp rx=0 tx=0).stop 2024-03-24T10:34:49.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 msgr2=0x7f22c415b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f22b800e760 tx=0x7f22b800ec20 comp rx=0 tx=0).stop 2024-03-24T10:34:49.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 shutdown_connections 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22b006b840 0x7f22b006dd00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22c4160c40 0x7f22c41610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22c415b030 0x7f22c415b490 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 --2- 172.21.15.1:0/655633394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22ac003970 0x7f22c415eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 >> 172.21.15.1:0/655633394 conn(0x7f22c40d3ec0 msgr2=0x7f22c40d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 shutdown_connections 2024-03-24T10:34:49.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.572+0000 7f22cb1fd640 1 -- 172.21.15.1:0/655633394 wait complete. 2024-03-24T10:34:49.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-24T10:34:49.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.686+0000 7f0b9b1b5640 1 Processor -- start 2024-03-24T10:34:49.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.686+0000 7f0b9b1b5640 1 -- start start 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b9b1b5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0b940d2bb0 0x7f0b940b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b9b1b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b9415e770 con 0x7f0b9415e390 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b9b1b5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b940d4e40 con 0x7f0b940d1000 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b9b1b5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f0b940b3190 con 0x7f0b940d2bb0 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b98f51640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b9415e390 0x7f0b940d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46870/0 (socket says 172.21.15.1:46870) 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.687+0000 7f0b98f51640 1 -- 172.21.15.1:0/2204841780 learned_addr learned my addr 172.21.15.1:0/2204841780 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023376544 0 0) 0x7f0b8c001490 con 0x7f0b9415e390 2024-03-24T10:34:49.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b7c003520 con 0x7f0b9415e390 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b93fff640 1 --2- 172.21.15.1:0/2204841780 >> v2:172.21.15.3:3300/0 conn(0x7f0b940d2bb0 0x7f0b940b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1838103800 0 0) 0x7f0b7c003520 con 0x7f0b9415e390 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 >> v2:172.21.15.3:3300/0 conn(0x7f0b940d2bb0 msgr2=0x7f0b940b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2204841780 >> v2:172.21.15.3:3300/0 conn(0x7f0b940d2bb0 0x7f0b940b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 >> v1:172.21.15.3:6789/0 conn(0x7f0b940d1000 legacy=0x7f0b940b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b940d1e40 con 0x7f0b9415e390 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f0b8c001b80 con 0x7f0b9415e390 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 >> v1:172.21.15.1:6789/0 conn(0x7f0b9415e390 legacy=0x7f0b940d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 0x7f0b7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b7c0047c0 0x7f0b7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c0052a0 0x7f0b7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c00c140 con 0x7f0b7c003970 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c00c2b0 con 0x7f0b7c0052a0 2024-03-24T10:34:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7c00c470 con 0x7f0b7c0047c0 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 0x7f0b7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b93fff640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b7c0047c0 0x7f0b7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b99752640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c0052a0 0x7f0b7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 -- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b7c0047c0 msgr2=0x7f0b7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b7c0047c0 0x7f0b7c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 -- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c0052a0 msgr2=0x7f0b7c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c0052a0 0x7f0b7c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b98f51640 1 -- 172.21.15.1:0/2204841780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7c003520 con 0x7f0b7c003970 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.688+0000 7f0b99752640 1 -- 172.21.15.1:0/2204841780 reap_dead start 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.689+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 0x7f0b7c003d90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00a800 tx=0x7f0b7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb72d15bf06af09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.689+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c034630 con 0x7f0b7c003970 2024-03-24T10:34:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.689+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c02f070 con 0x7f0b7c003970 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.689+0000 7f0b937fe640 1 -- 172.21.15.1:0/2204841780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c034a30 con 0x7f0b7c003970 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 msgr2=0x7f0b7c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 0x7f0b7c003d90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00a800 tx=0x7f0b7c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2204841780 shutdown_connections 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2204841780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b7c003970 0x7f0b7c003d90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2204841780 >> 172.21.15.1:0/2204841780 conn(0x7f0b940d3ec0 msgr2=0x7f0b940d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2204841780 shutdown_connections 2024-03-24T10:34:49.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.690+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2204841780 wait complete. 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.691+0000 7f0b9b1b5640 1 Processor -- start 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.691+0000 7f0b9b1b5640 1 -- start start 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 0x7f0b94153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b9409e090 0x7f0b94154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9415ae00 con 0x7f0b9409dc10 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0b94154390 con 0x7f0b7c003970 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b9b1b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0b940b47f0 con 0x7f0b9409e090 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52862/0 (socket says 172.21.15.1:52862) 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 -- 172.21.15.1:0/2168780988 learned_addr learned my addr 172.21.15.1:0/2168780988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 0x7f0b94153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 -- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b9409e090 msgr2=0x7f0b94154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b99752640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b9409e090 0x7f0b94154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b9409e090 0x7f0b94154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 -- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 msgr2=0x7f0b94153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 0x7f0b94153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b93fff640 1 -- 172.21.15.1:0/2168780988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b940b4ad0 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.692+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 0x7f0b94153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b93fff640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0b880119f0 tx=0x7f0b88011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88009660 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b940b4db0 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b940b5370 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8800fb50 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.693+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88014440 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.694+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b74005290 con 0x7f0b9409dc10 2024-03-24T10:34:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.694+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b880145a0 con 0x7f0b9409dc10 2024-03-24T10:34:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.695+0000 7f0b917fa640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 0x7f0b8006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.695+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f0b8808a8e0 con 0x7f0b9409dc10 2024-03-24T10:34:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.695+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 0x7f0b8006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:49.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.695+0000 7f0b98f51640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 0x7f0b8006dd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c034000 tx=0x7f0b8c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:49.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.699+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b880555b0 con 0x7f0b9409dc10 2024-03-24T10:34:49.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.948+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f0b74005ac0 con 0x7f0b9409dc10 2024-03-24T10:34:49.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.950+0000 7f0b917fa640 1 -- 172.21.15.1:0/2168780988 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f0b88058c70 con 0x7f0b9409dc10 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 msgr2=0x7f0b8006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 0x7f0b8006dd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c034000 tx=0x7f0b8c02f000 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 msgr2=0x7f0b94154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0b880119f0 tx=0x7f0b88011eb0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 shutdown_connections 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0b8006b840 0x7f0b8006dd00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0b9409e090 0x7f0b94154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b9409dc10 0x7f0b94154780 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 --2- 172.21.15.1:0/2168780988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c003970 0x7f0b94153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 >> 172.21.15.1:0/2168780988 conn(0x7f0b940d3ec0 msgr2=0x7f0b9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 shutdown_connections 2024-03-24T10:34:49.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:49.952+0000 7f0b9b1b5640 1 -- 172.21.15.1:0/2168780988 wait complete. 2024-03-24T10:34:49.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-24T10:34:50.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.064+0000 7fefce43b640 1 Processor -- start 2024-03-24T10:34:50.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefce43b640 1 -- start start 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefce43b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fefc80b31b0 0x7fefc8070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefce43b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc80cdd60 con 0x7fefc8144770 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefce43b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc806e200 con 0x7fefc80d00f0 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefce43b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fefc806eaa0 con 0x7fefc80b31b0 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefcc9d8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fefc80b31b0 0x7fefc8070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefc7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fefc8144770 0x7fefc806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46874/0 (socket says 172.21.15.1:46874) 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.065+0000 7fefc7fff640 1 -- 172.21.15.1:0/768401764 learned_addr learned my addr 172.21.15.1:0/768401764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:50.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.066+0000 7fefcc9d8640 1 -- 172.21.15.1:0/768401764 >> v1:172.21.15.3:6789/0 conn(0x7fefc80d00f0 legacy=0x7fefc80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.066+0000 7fefcc9d8640 1 -- 172.21.15.1:0/768401764 >> v1:172.21.15.1:6789/0 conn(0x7fefc8144770 legacy=0x7fefc806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.066+0000 7fefcc9d8640 1 -- 172.21.15.1:0/768401764 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc806e4e0 con 0x7fefc80b31b0 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.066+0000 7fefcc9d8640 1 --2- 172.21.15.1:0/768401764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80b31b0 0x7fefc8070b90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009d90 tx=0x7fefc002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f686d43c41981e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefc6ffd640 1 -- 172.21.15.1:0/768401764 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc002cd00 con 0x7fefc80b31b0 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefc6ffd640 1 -- 172.21.15.1:0/768401764 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc002ce60 con 0x7fefc80b31b0 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefc6ffd640 1 -- 172.21.15.1:0/768401764 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0034660 con 0x7fefc80b31b0 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 -- 172.21.15.1:0/768401764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80b31b0 msgr2=0x7fefc8070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 --2- 172.21.15.1:0/768401764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80b31b0 0x7fefc8070b90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009d90 tx=0x7fefc002bf50 comp rx=0 tx=0).stop 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 -- 172.21.15.1:0/768401764 shutdown_connections 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 --2- 172.21.15.1:0/768401764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80b31b0 0x7fefc8070b90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 -- 172.21.15.1:0/768401764 >> 172.21.15.1:0/768401764 conn(0x7fefc80b17a0 msgr2=0x7fefc80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 -- 172.21.15.1:0/768401764 shutdown_connections 2024-03-24T10:34:50.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.067+0000 7fefce43b640 1 -- 172.21.15.1:0/768401764 wait complete. 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.068+0000 7fefce43b640 1 Processor -- start 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 -- start start 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc80b31b0 0x7fefc813d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fefc8144770 0x7fefc8137780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8142340 con 0x7fefc80d00f0 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8142610 con 0x7fefc80b31b0 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefce43b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fefc81428f0 con 0x7fefc8144770 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52884/0 (socket says 172.21.15.1:52884) 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefc77fe640 1 -- 172.21.15.1:0/1505912710 learned_addr learned my addr 172.21.15.1:0/1505912710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:50.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefc7fff640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc80b31b0 0x7fefc813d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.069+0000 7fefcc9d8640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fefc8144770 0x7fefc8137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 -- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fefc8144770 msgr2=0x7fefc8137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fefc8144770 0x7fefc8137780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 -- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc80b31b0 msgr2=0x7fefc813d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc80b31b0 0x7fefc813d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 -- 172.21.15.1:0/1505912710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc0009a70 con 0x7fefc80d00f0 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc77fe640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c400 tx=0x7fefbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0052d0 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc005450 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc8142bd0 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc007260 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.070+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc8143190 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.071+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefc804f4e0 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.071+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fefbc0073c0 con 0x7fefc80d00f0 2024-03-24T10:34:50.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.072+0000 7fefc4ff9640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 0x7fefb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.072+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fefbc089290 con 0x7fefc80d00f0 2024-03-24T10:34:50.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.072+0000 7fefc7fff640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 0x7fefb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.072+0000 7fefc7fff640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 0x7fefb406dcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fefb8005880 tx=0x7fefb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.076+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fefbc054010 con 0x7fefc80d00f0 2024-03-24T10:34:50.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.328+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fefc80725e0 con 0x7fefc80d00f0 2024-03-24T10:34:50.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.330+0000 7fefc4ff9640 1 -- 172.21.15.1:0/1505912710 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7fefbc0576d0 con 0x7fefc80d00f0 2024-03-24T10:34:50.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 msgr2=0x7fefb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 0x7fefb406dcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fefb8005880 tx=0x7fefb8009040 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 msgr2=0x7fefc8141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c400 tx=0x7fefbc00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 shutdown_connections 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fefb406b810 0x7fefb406dcd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fefc8144770 0x7fefc8137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fefc80d00f0 0x7fefc8141e00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 --2- 172.21.15.1:0/1505912710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc80b31b0 0x7fefc813d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 >> 172.21.15.1:0/1505912710 conn(0x7fefc80b17a0 msgr2=0x7fefc8070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 shutdown_connections 2024-03-24T10:34:50.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.332+0000 7fefce43b640 1 -- 172.21.15.1:0/1505912710 wait complete. 2024-03-24T10:34:50.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:50.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.444+0000 7fa0fcc33640 1 Processor -- start 2024-03-24T10:34:50.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.444+0000 7fa0fcc33640 1 -- start start 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0fcc33640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0f806b7b0 0x7fa0f80b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0fcc33640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f8072c10 con 0x7fa0f815e0a0 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0fcc33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f806dde0 con 0x7fa0f8072710 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0fcc33640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f806e0c0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0f5d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0f806b7b0 0x7fa0f80b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0f5d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0f806b7b0 0x7fa0f80b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52894/0 (socket says 172.21.15.1:52894) 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0f659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0f815e0a0 0x7fa0f806b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46888/0 (socket says 172.21.15.1:46888) 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.445+0000 7fa0f5d9b640 1 -- 172.21.15.1:0/3672951833 learned_addr learned my addr 172.21.15.1:0/3672951833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.446+0000 7fa0f5d9b640 1 -- 172.21.15.1:0/3672951833 >> v1:172.21.15.3:6789/0 conn(0x7fa0f8072710 legacy=0x7fa0f80b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.446+0000 7fa0f5d9b640 1 -- 172.21.15.1:0/3672951833 >> v1:172.21.15.1:6789/0 conn(0x7fa0f815e0a0 legacy=0x7fa0f806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.446+0000 7fa0f5d9b640 1 -- 172.21.15.1:0/3672951833 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f806e3a0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.446+0000 7fa0f559a640 1 -- 172.21.15.1:0/3672951833 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744822734 0 0) 0x7fa0f8072c10 con 0x7fa0f815e0a0 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0f5d9b640 1 --2- 172.21.15.1:0/3672951833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b4380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009870 tx=0x7fa0ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c33b63b9d35cd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0f559a640 1 -- 172.21.15.1:0/3672951833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec030860 con 0x7fa0f806b7b0 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0f559a640 1 -- 172.21.15.1:0/3672951833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0ec0309c0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0f559a640 1 -- 172.21.15.1:0/3672951833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec030d50 con 0x7fa0f806b7b0 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 -- 172.21.15.1:0/3672951833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 msgr2=0x7fa0f80b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/3672951833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b4380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009870 tx=0x7fa0ec02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 -- 172.21.15.1:0/3672951833 shutdown_connections 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/3672951833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b4380 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 -- 172.21.15.1:0/3672951833 >> 172.21.15.1:0/3672951833 conn(0x7fa0f80b2820 msgr2=0x7fa0f80b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 -- 172.21.15.1:0/3672951833 shutdown_connections 2024-03-24T10:34:50.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.447+0000 7fa0fcc33640 1 -- 172.21.15.1:0/3672951833 wait complete. 2024-03-24T10:34:50.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 Processor -- start 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 -- start start 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0f8072710 0x7fa0f81f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f815e0a0 0x7fa0f81f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f814ff90 con 0x7fa0f806b7b0 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f815d2f0 con 0x7fa0f815e0a0 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0fcc33640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f80b1200 con 0x7fa0f8072710 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.449+0000 7fa0f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52898/0 (socket says 172.21.15.1:52898) 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 -- 172.21.15.1:0/4013956892 learned_addr learned my addr 172.21.15.1:0/4013956892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f6d9d640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f815e0a0 0x7fa0f81f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f5d9b640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0f8072710 0x7fa0f81f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 -- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0f8072710 msgr2=0x7fa0f81f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0f8072710 0x7fa0f81f4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 -- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f815e0a0 msgr2=0x7fa0f81f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f815e0a0 0x7fa0f81f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 -- 172.21.15.1:0/4013956892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec0094d0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0f659c640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c890 tx=0x7fa0f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000a640 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f000a7c0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f80b0220 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000ab70 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.450+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f80b2320 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.451+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f804f4e0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.452+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0f000acd0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.452+0000 7fa0e6ffd640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 0x7fa0d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.452+0000 7fa0f5d9b640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 0x7fa0d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.452+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fa0f00896b0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.453+0000 7fa0f5d9b640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 0x7fa0d806dcd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec036000 tx=0x7fa0ec03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.457+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa0f00544b0 con 0x7fa0f806b7b0 2024-03-24T10:34:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.640+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0f81f4970 con 0x7fa0f806b7b0 2024-03-24T10:34:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.640+0000 7fa0e6ffd640 1 -- 172.21.15.1:0/4013956892 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+12759 (secure 0 0 0) 0x7fa0f0018090 con 0x7fa0f806b7b0 2024-03-24T10:34:50.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:50.641 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":30,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:47.662807+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 msgr2=0x7fa0d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 0x7fa0d806dcd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec036000 tx=0x7fa0ec03b040 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 msgr2=0x7fa0f80b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c890 tx=0x7fa0f000cd50 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 shutdown_connections 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa0d806b810 0x7fa0d806dcd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0f815e0a0 0x7fa0f81f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa0f8072710 0x7fa0f81f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 --2- 172.21.15.1:0/4013956892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0f806b7b0 0x7fa0f80b0cc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 >> 172.21.15.1:0/4013956892 conn(0x7fa0f80b2820 msgr2=0x7fa0f80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 shutdown_connections 2024-03-24T10:34:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.642+0000 7fa0fcc33640 1 -- 172.21.15.1:0/4013956892 wait complete. 2024-03-24T10:34:50.653 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:34:50.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:34:50.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.758+0000 7fc2a9240640 1 Processor -- start 2024-03-24T10:34:50.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.758+0000 7fc2a9240640 1 -- start start 2024-03-24T10:34:50.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a9240640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2a406e5b0 0x7fc2a406e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a9240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a415c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a9240640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a406eed0 con 0x7fc2a40b4570 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a9240640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a40b34d0 con 0x7fc2a406e5b0 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a9240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a40b37b0 con 0x7fc2a415c4d0 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a359e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a415c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a2d9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2a406e5b0 0x7fc2a406e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a259c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc2a40b4570 0x7fc2a40b2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49814/0 (socket says 172.21.15.1:49814) 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.759+0000 7fc2a259c640 1 -- 172.21.15.1:0/2691866538 learned_addr learned my addr 172.21.15.1:0/2691866538 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a359e640 1 -- 172.21.15.1:0/2691866538 >> v2:172.21.15.3:3300/0 conn(0x7fc2a406e5b0 msgr2=0x7fc2a406e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a1d9b640 1 -- 172.21.15.1:0/2691866538 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807799291 0 0) 0x7fc2a406eed0 con 0x7fc2a40b4570 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a359e640 1 --2- 172.21.15.1:0/2691866538 >> v2:172.21.15.3:3300/0 conn(0x7fc2a406e5b0 0x7fc2a406e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a359e640 1 -- 172.21.15.1:0/2691866538 >> v1:172.21.15.3:6789/0 conn(0x7fc2a40b4570 legacy=0x7fc2a40b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a359e640 1 -- 172.21.15.1:0/2691866538 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a40b3a90 con 0x7fc2a415c4d0 2024-03-24T10:34:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.760+0000 7fc2a359e640 1 --2- 172.21.15.1:0/2691866538 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a415c960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0097a0 tx=0x7fc29c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4244d374627b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a1d9b640 1 -- 172.21.15.1:0/2691866538 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c031830 con 0x7fc2a415c4d0 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a1d9b640 1 -- 172.21.15.1:0/2691866538 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29c031990 con 0x7fc2a415c4d0 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a1d9b640 1 -- 172.21.15.1:0/2691866538 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c031d20 con 0x7fc2a415c4d0 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 -- 172.21.15.1:0/2691866538 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 msgr2=0x7fc2a415c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2691866538 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a415c960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0097a0 tx=0x7fc29c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 -- 172.21.15.1:0/2691866538 shutdown_connections 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2691866538 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a415c960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2691866538 >> v2:172.21.15.3:3300/0 conn(0x7fc2a406e5b0 0x7fc2a406e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 -- 172.21.15.1:0/2691866538 >> 172.21.15.1:0/2691866538 conn(0x7fc2a40b1710 msgr2=0x7fc2a40b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 -- 172.21.15.1:0/2691866538 shutdown_connections 2024-03-24T10:34:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.761+0000 7fc2a9240640 1 -- 172.21.15.1:0/2691866538 wait complete. 2024-03-24T10:34:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 Processor -- start 2024-03-24T10:34:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 -- start start 2024-03-24T10:34:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a40b4570 0x7fc2a413a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a413ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a40cf0b0 con 0x7fc2a406e5b0 2024-03-24T10:34:50.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a415a720 con 0x7fc2a40b4570 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a9240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a4138d80 con 0x7fc2a415c4d0 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47428/0 (socket says 172.21.15.1:47428) 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a2d9d640 1 -- 172.21.15.1:0/2865212048 learned_addr learned my addr 172.21.15.1:0/2865212048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a259c640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a40b4570 0x7fc2a413a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.763+0000 7fc2a359e640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a413ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 -- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 msgr2=0x7fc2a413ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a413ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 -- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a40b4570 msgr2=0x7fc2a413a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a40b4570 0x7fc2a413a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 -- 172.21.15.1:0/2865212048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29c009400 con 0x7fc2a406e5b0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a2d9d640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc29400efc0 tx=0x7fc29400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294009b80 con 0x7fc2a406e5b0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29400f040 con 0x7fc2a406e5b0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a413bfa0 con 0x7fc2a406e5b0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2940146a0 con 0x7fc2a406e5b0 2024-03-24T10:34:50.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.764+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a413c280 con 0x7fc2a406e5b0 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.765+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a40cf320 con 0x7fc2a406e5b0 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.766+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc294014800 con 0x7fc2a406e5b0 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.766+0000 7fc29b7fe640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 0x7fc28406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.766+0000 7fc2a259c640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 0x7fc28406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.766+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fc29408a860 con 0x7fc2a406e5b0 2024-03-24T10:34:50.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.767+0000 7fc2a259c640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 0x7fc28406dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4144140 tx=0x7fc28c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:50.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.771+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc294055680 con 0x7fc2a406e5b0 2024-03-24T10:34:50.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.951+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc2a404f4e0 con 0x7fc2a406e5b0 2024-03-24T10:34:50.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.951+0000 7fc29b7fe640 1 -- 172.21.15.1:0/2865212048 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7fc294058d40 con 0x7fc2a406e5b0 2024-03-24T10:34:50.952 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:34:50.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 msgr2=0x7fc28406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 0x7fc28406dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4144140 tx=0x7fc28c007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 msgr2=0x7fc2a41ef9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc29400efc0 tx=0x7fc29400c490 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 shutdown_connections 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc28406b810 0x7fc28406dcd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc2a415c4d0 0x7fc2a413ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a40b4570 0x7fc2a413a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 --2- 172.21.15.1:0/2865212048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a406e5b0 0x7fc2a41ef9e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 >> 172.21.15.1:0/2865212048 conn(0x7fc2a40b1710 msgr2=0x7fc2a40cbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 shutdown_connections 2024-03-24T10:34:50.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:50.953+0000 7fc2a9240640 1 -- 172.21.15.1:0/2865212048 wait complete. 2024-03-24T10:34:50.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T10:34:51.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.065+0000 7f6574ba0640 1 Processor -- start 2024-03-24T10:34:51.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 -- start start 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f657015e390 0x7f65700d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 0x7f65700d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65700b0b70 con 0x7f65700b2ed0 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f65700d1d60 con 0x7f657015e390 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.066+0000 7f6574ba0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f65700b0de0 con 0x7f65700cfae0 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656ed9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 0x7f65700d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f657015e390 0x7f65700d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f6567fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f65700b2ed0 0x7f65700b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49822/0 (socket says 172.21.15.1:49822) 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f6567fff640 1 -- 172.21.15.1:0/676822882 learned_addr learned my addr 172.21.15.1:0/676822882 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656e59c640 1 -- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 msgr2=0x7f65700d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656e59c640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 0x7f65700d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656e59c640 1 -- 172.21.15.1:0/676822882 >> v1:172.21.15.3:6789/0 conn(0x7f65700b2ed0 legacy=0x7f65700b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656e59c640 1 -- 172.21.15.1:0/676822882 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657015e770 con 0x7f657015e390 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656dd9b640 1 -- 172.21.15.1:0/676822882 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267274034 0 0) 0x7f65700b0b70 con 0x7f65700b2ed0 2024-03-24T10:34:51.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.067+0000 7f656ed9d640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 0x7f65700d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f656e59c640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f657015e390 0x7f65700d2310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=806d8ccd2eb9e989 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f656dd9b640 1 -- 172.21.15.1:0/676822882 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560031830 con 0x7f657015e390 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f656dd9b640 1 -- 172.21.15.1:0/676822882 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6560031990 con 0x7f657015e390 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f656dd9b640 1 -- 172.21.15.1:0/676822882 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560031d20 con 0x7f657015e390 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f6574ba0640 1 -- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f657015e390 msgr2=0x7f65700d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.068+0000 7f6574ba0640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f657015e390 0x7f65700d2310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f65600097a0 tx=0x7f656002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 -- 172.21.15.1:0/676822882 shutdown_connections 2024-03-24T10:34:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f65700cfae0 0x7f65700d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 --2- 172.21.15.1:0/676822882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f657015e390 0x7f65700d2310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 -- 172.21.15.1:0/676822882 >> 172.21.15.1:0/676822882 conn(0x7f65700d3ee0 msgr2=0x7f65700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 -- 172.21.15.1:0/676822882 shutdown_connections 2024-03-24T10:34:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.069+0000 7f6574ba0640 1 -- 172.21.15.1:0/676822882 wait complete. 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.070+0000 7f6574ba0640 1 Processor -- start 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.070+0000 7f6574ba0640 1 -- start start 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65700b2ed0 0x7f65701f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f657015e390 0x7f65700b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f65701f7050 con 0x7f65700cfae0 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f65701f7330 con 0x7f65700b2ed0 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f65701f7610 con 0x7f657015e390 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47434/0 (socket says 172.21.15.1:47434) 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 -- 172.21.15.1:0/524214611 learned_addr learned my addr 172.21.15.1:0/524214611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f656e59c640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65700b2ed0 0x7f65701f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f656ed9d640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f657015e390 0x7f65700b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 -- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f657015e390 msgr2=0x7f65700b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f657015e390 0x7f65700b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 -- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65700b2ed0 msgr2=0x7f65701f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65700b2ed0 0x7f65701f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 -- 172.21.15.1:0/524214611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6568009a70 con 0x7f65700cfae0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6567fff640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f655800c4d0 tx=0x7f655800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65580074e0 con 0x7f65700cfae0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6558007640 con 0x7f65700cfae0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6560009400 con 0x7f65700cfae0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558004a20 con 0x7f65700cfae0 2024-03-24T10:34:51.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.071+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65701f7bd0 con 0x7f65700cfae0 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.072+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657004f4e0 con 0x7f65700cfae0 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.073+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6558004b80 con 0x7f65700cfae0 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.073+0000 7f6566ffd640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 0x7f655006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.073+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f6558089ca0 con 0x7f65700cfae0 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.073+0000 7f656e59c640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 0x7f655006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.074+0000 7f656e59c640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 0x7f655006dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6560036000 tx=0x7f656003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.078+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6558054990 con 0x7f65700cfae0 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.262+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f65700d3210 con 0x7f65700cfae0 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.262+0000 7f6566ffd640 1 -- 172.21.15.1:0/524214611 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) ==== 84+0+65 (secure 0 0 0) 0x7f6558058050 con 0x7f65700cfae0 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:34:51.263 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 msgr2=0x7f655006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 0x7f655006dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6560036000 tx=0x7f656003d040 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 msgr2=0x7f6570161140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f655800c4d0 tx=0x7f655800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 shutdown_connections 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f655006b810 0x7f655006dcd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f657015e390 0x7f65700b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700cfae0 0x7f6570161140 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 --2- 172.21.15.1:0/524214611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65700b2ed0 0x7f65701f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 >> 172.21.15.1:0/524214611 conn(0x7f65700d3ee0 msgr2=0x7f657006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 shutdown_connections 2024-03-24T10:34:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.264+0000 7f6574ba0640 1 -- 172.21.15.1:0/524214611 wait complete. 2024-03-24T10:34:51.275 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T10:34:51.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T10:34:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.381+0000 7fde12d6c640 1 Processor -- start 2024-03-24T10:34:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde12d6c640 1 -- start start 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde12d6c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde0c0cdc70 0x7fde0c0afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde12d6c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0b0040 con 0x7fde0c15bed0 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde12d6c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0ce250 con 0x7fde0c0b5880 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde12d6c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fde0c15c2b0 con 0x7fde0c0cdc70 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde0bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde0c0cdc70 0x7fde0c0afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde10b08640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde0c15bed0 0x7fde0c15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60872/0 (socket says 172.21.15.1:60872) 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde10b08640 1 -- 172.21.15.1:0/4157179281 learned_addr learned my addr 172.21.15.1:0/4157179281 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.382+0000 7fde0bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde0c0cdc70 0x7fde0c0afb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47442/0 (socket says 172.21.15.1:47442) 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.383+0000 7fde0bfff640 1 -- 172.21.15.1:0/4157179281 >> v1:172.21.15.3:6789/0 conn(0x7fde0c0b5880 legacy=0x7fde0c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:51.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.383+0000 7fde0bfff640 1 -- 172.21.15.1:0/4157179281 >> v1:172.21.15.1:6789/0 conn(0x7fde0c15bed0 legacy=0x7fde0c15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.383+0000 7fde0bfff640 1 -- 172.21.15.1:0/4157179281 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c0ce530 con 0x7fde0c0cdc70 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.383+0000 7fde0bfff640 1 --2- 172.21.15.1:0/4157179281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0cdc70 0x7fde0c0afb00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097a0 tx=0x7fddf802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea15c862cfda1638 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde0b7fe640 1 -- 172.21.15.1:0/4157179281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8031830 con 0x7fde0c0cdc70 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde0b7fe640 1 -- 172.21.15.1:0/4157179281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf8031990 con 0x7fde0c0cdc70 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde0b7fe640 1 -- 172.21.15.1:0/4157179281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8031d20 con 0x7fde0c0cdc70 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 -- 172.21.15.1:0/4157179281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0cdc70 msgr2=0x7fde0c0afb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 --2- 172.21.15.1:0/4157179281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0cdc70 0x7fde0c0afb00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097a0 tx=0x7fddf802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 -- 172.21.15.1:0/4157179281 shutdown_connections 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 --2- 172.21.15.1:0/4157179281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0cdc70 0x7fde0c0afb00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 -- 172.21.15.1:0/4157179281 >> 172.21.15.1:0/4157179281 conn(0x7fde0c072410 msgr2=0x7fde0c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 -- 172.21.15.1:0/4157179281 shutdown_connections 2024-03-24T10:34:51.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.384+0000 7fde12d6c640 1 -- 172.21.15.1:0/4157179281 wait complete. 2024-03-24T10:34:51.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.385+0000 7fde12d6c640 1 Processor -- start 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.385+0000 7fde12d6c640 1 -- start start 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fde0c0cdc70 0x7fde0c0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 0x7fde0c0b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0ad4d0 con 0x7fde0c0b5880 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0ad7b0 con 0x7fde0c15bed0 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde12d6c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fde0c0ada90 con 0x7fde0c0cdc70 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde10b08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde10b08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47448/0 (socket says 172.21.15.1:47448) 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.386+0000 7fde10b08640 1 -- 172.21.15.1:0/2142499338 learned_addr learned my addr 172.21.15.1:0/2142499338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde0bfff640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fde0c0cdc70 0x7fde0c0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde11309640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 0x7fde0c0b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 -- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fde0c0cdc70 msgr2=0x7fde0c0b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fde0c0cdc70 0x7fde0c0b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 -- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 msgr2=0x7fde0c0b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 0x7fde0c0b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 -- 172.21.15.1:0/2142499338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf8009400 con 0x7fde0c0b5880 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde11309640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 0x7fde0c0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde10b08640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c3c0 tx=0x7fde0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00005ce0 con 0x7fde0c0b5880 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde00005e40 con 0x7fde0c0b5880 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c0add70 con 0x7fde0c0b5880 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde000072d0 con 0x7fde0c0b5880 2024-03-24T10:34:51.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.387+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0c0aa320 con 0x7fde0c0b5880 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.388+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0c04f470 con 0x7fde0c0b5880 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.388+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fde000074f0 con 0x7fde0c0b5880 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.389+0000 7fde097fa640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 0x7fddf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.389+0000 7fde0bfff640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 0x7fddf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.389+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fde000892a0 con 0x7fde0c0b5880 2024-03-24T10:34:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.389+0000 7fde0bfff640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 0x7fddf006dcd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036000 tx=0x7fddf8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:51.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.393+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fde00054010 con 0x7fde0c0b5880 2024-03-24T10:34:51.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:51.573+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fde0c0b0040 con 0x7fde0c0b5880 2024-03-24T10:34:52.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.333+0000 7fde097fa640 1 -- 172.21.15.1:0/2142499338 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) ==== 138+0+0 (secure 0 0 0) 0x7fde0c0b0040 con 0x7fde0c0b5880 2024-03-24T10:34:52.334 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-24T10:34:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 msgr2=0x7fddf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 0x7fddf006dcd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036000 tx=0x7fddf8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 msgr2=0x7fde0c155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c3c0 tx=0x7fde0000c880 comp rx=0 tx=0).stop 2024-03-24T10:34:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 shutdown_connections 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fddf006b810 0x7fddf006dcd0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036000 tx=0x7fddf8008030 comp rx=0 tx=0).stop 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde0c15bed0 0x7fde0c0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fde0c0cdc70 0x7fde0c0b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 --2- 172.21.15.1:0/2142499338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde0c0b5880 0x7fde0c155f50 secure :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c3c0 tx=0x7fde0000c880 comp rx=0 tx=0).stop 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.339+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 >> 172.21.15.1:0/2142499338 conn(0x7fde0c072410 msgr2=0x7fde0c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.340+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 shutdown_connections 2024-03-24T10:34:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.340+0000 7fde12d6c640 1 -- 172.21.15.1:0/2142499338 wait complete. 2024-03-24T10:34:52.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T10:34:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.469+0000 7f6154095640 1 Processor -- start 2024-03-24T10:34:52.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.469+0000 7f6154095640 1 -- start start 2024-03-24T10:34:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6154095640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f614c0b31d0 0x7f614c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6154095640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f614c06d240 0x7f614c0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6154095640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614c0cc250 con 0x7f614c0d0e90 2024-03-24T10:34:52.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6154095640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f614c0b35b0 con 0x7f614c0b31d0 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6154095640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f614c0cc4a0 con 0x7f614c06d240 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f614c0b31d0 0x7f614c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f614c0b31d0 0x7f614c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47456/0 (socket says 172.21.15.1:47456) 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 -- 172.21.15.1:0/499591558 learned_addr learned my addr 172.21.15.1:0/499591558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 -- 172.21.15.1:0/499591558 >> v2:172.21.15.3:3300/0 conn(0x7f614c06d240 msgr2=0x7f614c0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 --2- 172.21.15.1:0/499591558 >> v2:172.21.15.3:3300/0 conn(0x7f614c06d240 0x7f614c0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 -- 172.21.15.1:0/499591558 >> v1:172.21.15.3:6789/0 conn(0x7f614c0d0e90 legacy=0x7f614c06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:52.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.470+0000 7f6151e31640 1 -- 172.21.15.1:0/499591558 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c0cc780 con 0x7f614c0b31d0 2024-03-24T10:34:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6151e31640 1 --2- 172.21.15.1:0/499591558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c0d0950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f614c0c56e0 tx=0x7f614802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232569b24de74acd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6150e2f640 1 -- 172.21.15.1:0/499591558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148031870 con 0x7f614c0b31d0 2024-03-24T10:34:52.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6150e2f640 1 -- 172.21.15.1:0/499591558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61480319d0 con 0x7f614c0b31d0 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6150e2f640 1 -- 172.21.15.1:0/499591558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148031d60 con 0x7f614c0b31d0 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6154095640 1 -- 172.21.15.1:0/499591558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 msgr2=0x7f614c0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.471+0000 7f6154095640 1 --2- 172.21.15.1:0/499591558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c0d0950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f614c0c56e0 tx=0x7f614802b910 comp rx=0 tx=0).stop 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 -- 172.21.15.1:0/499591558 shutdown_connections 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 --2- 172.21.15.1:0/499591558 >> v2:172.21.15.3:3300/0 conn(0x7f614c06d240 0x7f614c0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 --2- 172.21.15.1:0/499591558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c0d0950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 -- 172.21.15.1:0/499591558 >> 172.21.15.1:0/499591558 conn(0x7f614c0d1c00 msgr2=0x7f614c0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 -- 172.21.15.1:0/499591558 shutdown_connections 2024-03-24T10:34:52.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.472+0000 7f6154095640 1 -- 172.21.15.1:0/499591558 wait complete. 2024-03-24T10:34:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 Processor -- start 2024-03-24T10:34:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 -- start start 2024-03-24T10:34:52.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06d240 0x7f614c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f614c0d0e90 0x7f614c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0c9fd0 con 0x7f614c0b31d0 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0c8430 con 0x7f614c06d240 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.473+0000 7f6154095640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f614c0c8cd0 con 0x7f614c0d0e90 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47458/0 (socket says 172.21.15.1:47458) 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 -- 172.21.15.1:0/1130146179 learned_addr learned my addr 172.21.15.1:0/1130146179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151e31640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06d240 0x7f614c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6152632640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f614c0d0e90 0x7f614c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 -- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f614c0d0e90 msgr2=0x7f614c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f614c0d0e90 0x7f614c0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 -- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06d240 msgr2=0x7f614c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06d240 0x7f614c0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:52.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 -- 172.21.15.1:0/1130146179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61480094d0 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6151630640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b3c0 tx=0x7f613c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c011830 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c011990 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c0c8fb0 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c011d20 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.474+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c0c9570 con 0x7f614c0b31d0 2024-03-24T10:34:52.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.475+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c04f470 con 0x7f614c0b31d0 2024-03-24T10:34:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.476+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f613c01b460 con 0x7f614c0b31d0 2024-03-24T10:34:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.476+0000 7f61427fc640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 0x7f613006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.476+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5227+0+0 (secure 0 0 0) 0x7f613c089cb0 con 0x7f614c0b31d0 2024-03-24T10:34:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.476+0000 7f6151e31640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 0x7f613006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:52.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.477+0000 7f6151e31640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 0x7f613006dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6148036000 tx=0x7f614803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:52.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.481+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f613c054960 con 0x7f614c0b31d0 2024-03-24T10:34:52.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:52.661+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f614c06ef50 con 0x7f614c0b31d0 2024-03-24T10:34:53.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.339+0000 7f61427fc640 1 -- 172.21.15.1:0/1130146179 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) ==== 170+0+0 (secure 0 0 0) 0x7f613c058020 con 0x7f614c0b31d0 2024-03-24T10:34:53.339 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-24T10:34:53.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 msgr2=0x7f613006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 0x7f613006dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6148036000 tx=0x7f614803c040 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 msgr2=0x7f614c160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b3c0 tx=0x7f613c00b880 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 shutdown_connections 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f613006b810 0x7f613006dcd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f614c0d0e90 0x7f614c0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f614c0b31d0 0x7f614c160e30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 --2- 172.21.15.1:0/1130146179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614c06d240 0x7f614c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 >> 172.21.15.1:0/1130146179 conn(0x7f614c0d1c00 msgr2=0x7f614c0b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 shutdown_connections 2024-03-24T10:34:53.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.342+0000 7f6154095640 1 -- 172.21.15.1:0/1130146179 wait complete. 2024-03-24T10:34:53.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T10:34:53.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.476+0000 7f773a447640 1 Processor -- start 2024-03-24T10:34:53.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.476+0000 7f773a447640 1 -- start start 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7734144a10 0x7f773406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f77340cff00 0x7f77340d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f773415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7734160910 con 0x7f7734144a10 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f77340d08b0 con 0x7f77340cff00 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f773a447640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7734160630 con 0x7f773415f920 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f7733fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7734144a10 0x7f773406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f7733fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7734144a10 0x7f773406da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47470/0 (socket says 172.21.15.1:47470) 2024-03-24T10:34:53.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f7733fff640 1 -- 172.21.15.1:0/1913836000 learned_addr learned my addr 172.21.15.1:0/1913836000 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f77337fe640 1 --2- 172.21.15.1:0/1913836000 >> v2:172.21.15.3:3300/0 conn(0x7f77340cff00 0x7f77340d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.477+0000 7f77389e4640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f773415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 -- 172.21.15.1:0/1913836000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 msgr2=0x7f773415fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f773415fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 -- 172.21.15.1:0/1913836000 >> v2:172.21.15.3:3300/0 conn(0x7f77340cff00 msgr2=0x7f77340d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 --2- 172.21.15.1:0/1913836000 >> v2:172.21.15.3:3300/0 conn(0x7f77340cff00 0x7f77340d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 -- 172.21.15.1:0/1913836000 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773406cf80 con 0x7f7734144a10 2024-03-24T10:34:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7733fff640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7734144a10 0x7f773406da60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77340af720 tx=0x7f772802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1ee17f98565a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7732ffd640 1 -- 172.21.15.1:0/1913836000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728031640 con 0x7f7734144a10 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7732ffd640 1 -- 172.21.15.1:0/1913836000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280317a0 con 0x7f7734144a10 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.478+0000 7f7732ffd640 1 -- 172.21.15.1:0/1913836000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728031b30 con 0x7f7734144a10 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 -- 172.21.15.1:0/1913836000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7734144a10 msgr2=0x7f773406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7734144a10 0x7f773406da60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77340af720 tx=0x7f772802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 -- 172.21.15.1:0/1913836000 shutdown_connections 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f773415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 --2- 172.21.15.1:0/1913836000 >> v2:172.21.15.3:3300/0 conn(0x7f77340cff00 0x7f77340d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 --2- 172.21.15.1:0/1913836000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7734144a10 0x7f773406da60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 -- 172.21.15.1:0/1913836000 >> 172.21.15.1:0/1913836000 conn(0x7f773406b4f0 msgr2=0x7f773406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:53.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 -- 172.21.15.1:0/1913836000 shutdown_connections 2024-03-24T10:34:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.479+0000 7f773a447640 1 -- 172.21.15.1:0/1913836000 wait complete. 2024-03-24T10:34:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.480+0000 7f773a447640 1 Processor -- start 2024-03-24T10:34:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.480+0000 7f773a447640 1 -- start start 2024-03-24T10:34:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 0x7f7734143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f77341438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f773415c120 con 0x7f77340cff00 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f77341f86e0 con 0x7f7734144a10 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f773a447640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f77341f7940 con 0x7f773415f920 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47480/0 (socket says 172.21.15.1:47480) 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 -- 172.21.15.1:0/1878665255 learned_addr learned my addr 172.21.15.1:0/1878665255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f77389e4640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f77341438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f77337fe640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 0x7f7734143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 -- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 msgr2=0x7f77341438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f77341438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 -- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 msgr2=0x7f7734143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 0x7f7734143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.481+0000 7f7733fff640 1 -- 172.21.15.1:0/1878665255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77280094d0 con 0x7f77340cff00 2024-03-24T10:34:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f77389e4640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f77341438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f77337fe640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 0x7f7734143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f7733fff640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f77280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772803ac60 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f772803adc0 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7734144f70 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728044630 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.482+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77341f7ee0 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.483+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773404f470 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.483+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f772802f050 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.484+0000 7f7730ff9640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.484+0000 7f77337fe640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.484+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5609+0+0 (secure 0 0 0) 0x7f77280ac050 con 0x7f77340cff00 2024-03-24T10:34:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.484+0000 7f77337fe640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 0x7f772006dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77340b14b0 tx=0x7f7724008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.489+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7728076ad0 con 0x7f77340cff00 2024-03-24T10:34:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:53.708+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7734072ad0 con 0x7f77340cff00 2024-03-24T10:34:54.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.340+0000 7f7730ff9640 1 -- 172.21.15.1:0/1878665255 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f7728031e60 con 0x7f77340cff00 2024-03-24T10:34:54.341 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 4 and data pool 5 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.347+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 msgr2=0x7f772006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.347+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 0x7f772006dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77340b14b0 tx=0x7f7724008040 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.347+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 msgr2=0x7f773415bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.347+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f77280046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 shutdown_connections 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f773415f920 0x7f77341438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7734144a10 0x7f7734143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 --2- 172.21.15.1:0/1878665255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340cff00 0x7f773415bbe0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.348+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 >> 172.21.15.1:0/1878665255 conn(0x7f773406b4f0 msgr2=0x7f77340ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.349+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 shutdown_connections 2024-03-24T10:34:54.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.349+0000 7f773a447640 1 -- 172.21.15.1:0/1878665255 wait complete. 2024-03-24T10:34:54.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.488+0000 7f4fa6855640 1 Processor -- start 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.488+0000 7f4fa6855640 1 -- start start 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa6855640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fa00cebc0 0x7f4fa00d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa6855640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa00d4eb0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa6855640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa00cf0e0 con 0x7f4fa00d2ca0 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa6855640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4fa00d0160 con 0x7f4fa00cebc0 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa4df2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fa00cebc0 0x7f4fa00d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4f9ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4fa015e0a0 0x7f4fa00d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60882/0 (socket says 172.21.15.1:60882) 2024-03-24T10:34:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa4df2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fa00cebc0 0x7f4fa00d4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47490/0 (socket says 172.21.15.1:47490) 2024-03-24T10:34:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4f9ffff640 1 -- 172.21.15.1:0/3818634961 learned_addr learned my addr 172.21.15.1:0/3818634961 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa4df2640 1 -- 172.21.15.1:0/3818634961 >> v1:172.21.15.3:6789/0 conn(0x7f4fa00d2ca0 legacy=0x7f4fa00b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa4df2640 1 -- 172.21.15.1:0/3818634961 >> v1:172.21.15.1:6789/0 conn(0x7f4fa015e0a0 legacy=0x7f4fa00d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4fa4df2640 1 -- 172.21.15.1:0/3818634961 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa00cf3c0 con 0x7f4fa00cebc0 2024-03-24T10:34:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.489+0000 7f4f9effd640 1 -- 172.21.15.1:0/3818634961 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454952576 0 0) 0x7f4f94001110 con 0x7f4fa015e0a0 2024-03-24T10:34:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.490+0000 7f4fa4df2640 1 --2- 172.21.15.1:0/3818634961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa00cebc0 0x7f4fa00d4970 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f9802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6353fc54b29456 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.490+0000 7f4f9effd640 1 -- 172.21.15.1:0/3818634961 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9802cd80 con 0x7f4fa00cebc0 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.490+0000 7f4f9effd640 1 -- 172.21.15.1:0/3818634961 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f98034440 con 0x7f4fa00cebc0 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.490+0000 7f4f9effd640 1 -- 172.21.15.1:0/3818634961 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f980347d0 con 0x7f4fa00cebc0 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 -- 172.21.15.1:0/3818634961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa00cebc0 msgr2=0x7f4fa00d4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 --2- 172.21.15.1:0/3818634961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa00cebc0 0x7f4fa00d4970 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f9802bfd0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 -- 172.21.15.1:0/3818634961 shutdown_connections 2024-03-24T10:34:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 --2- 172.21.15.1:0/3818634961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa00cebc0 0x7f4fa00d4970 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 -- 172.21.15.1:0/3818634961 >> 172.21.15.1:0/3818634961 conn(0x7f4fa009d1c0 msgr2=0x7f4fa00d3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 -- 172.21.15.1:0/3818634961 shutdown_connections 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.491+0000 7f4fa6855640 1 -- 172.21.15.1:0/3818634961 wait complete. 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.492+0000 7f4fa6855640 1 Processor -- start 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.492+0000 7f4fa6855640 1 -- start start 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fa00cebc0 0x7f4fa00b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fa00d2ca0 0x7f4fa00b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01f4b70 con 0x7f4fa015e0a0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01f4890 con 0x7f4fa00d2ca0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa6855640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4fa00b2f70 con 0x7f4fa00cebc0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47494/0 (socket says 172.21.15.1:47494) 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 -- 172.21.15.1:0/1859695053 learned_addr learned my addr 172.21.15.1:0/1859695053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4f9f7fe640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fa00d2ca0 0x7f4fa00b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4f9ffff640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fa00cebc0 0x7f4fa00b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 -- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fa00cebc0 msgr2=0x7f4fa00b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fa00cebc0 0x7f4fa00b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 -- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fa00d2ca0 msgr2=0x7f4fa00b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fa00d2ca0 0x7f4fa00b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 -- 172.21.15.1:0/1859695053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98009a70 con 0x7f4fa015e0a0 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.493+0000 7f4fa4df2640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f98037040 tx=0x7f4f98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.494+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f98003b90 con 0x7f4fa015e0a0 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.494+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f98003d10 con 0x7f4fa015e0a0 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.494+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa00b3250 con 0x7f4fa015e0a0 2024-03-24T10:34:54.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.494+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f98004270 con 0x7f4fa015e0a0 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.494+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa00b70a0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.495+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa00b2dd0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.495+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f980043d0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.495+0000 7f4f9cff9640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 0x7f4f8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.496+0000 7f4f9ffff640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 0x7f4f8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.496+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4f980accd0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.496+0000 7f4f9ffff640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 0x7f4f8c06dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4f94000f60 tx=0x7f4f94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.500+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4fa00b2dd0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.681+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fa00b2dd0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.682+0000 7f4f9cff9640 1 -- 172.21.15.1:0/1859695053 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16550 (secure 0 0 0) 0x7f4fa00b2dd0 con 0x7f4fa015e0a0 2024-03-24T10:34:54.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:54.683 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:53.709157+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 msgr2=0x7f4f8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 0x7f4f8c06dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4f94000f60 tx=0x7f4f94009040 comp rx=0 tx=0).stop 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 msgr2=0x7f4fa015d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f98037040 tx=0x7f4f98004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 shutdown_connections 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f8c06b810 0x7f4f8c06dcd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa015e0a0 0x7f4fa015d140 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fa00d2ca0 0x7f4fa00b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 --2- 172.21.15.1:0/1859695053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fa00cebc0 0x7f4fa00b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 >> 172.21.15.1:0/1859695053 conn(0x7f4fa009d1c0 msgr2=0x7f4fa00b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 shutdown_connections 2024-03-24T10:34:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.684+0000 7f4fa6855640 1 -- 172.21.15.1:0/1859695053 wait complete. 2024-03-24T10:34:54.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.797+0000 7ff895a0b640 1 Processor -- start 2024-03-24T10:34:54.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.797+0000 7ff895a0b640 1 -- start start 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff895a0b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff89006b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff895a0b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff890162a90 con 0x7ff8900b0180 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff895a0b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff890072c10 con 0x7ff890072710 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff895a0b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff89006dde0 con 0x7ff89015e0a0 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff8900b0180 0x7ff890162380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60884/0 (socket says 172.21.15.1:60884) 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88f7fe640 1 -- 172.21.15.1:0/3944452004 learned_addr learned my addr 172.21.15.1:0/3944452004 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:54.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88effd640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff89006b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208023594 0 0) 0x7ff890162a90 con 0x7ff8900b0180 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff87c003520 con 0x7ff8900b0180 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3960150102 0 0) 0x7ff87c003520 con 0x7ff8900b0180 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 msgr2=0x7ff89006b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff89006b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 >> v1:172.21.15.3:6789/0 conn(0x7ff890072710 legacy=0x7ff890161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89006e0c0 con 0x7ff8900b0180 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7ff8880019f0 con 0x7ff8900b0180 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 >> v1:172.21.15.1:6789/0 conn(0x7ff8900b0180 legacy=0x7ff890162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff87c0039a0 0x7ff87c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff87c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c004d00 0x7ff87c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff89006dde0 con 0x7ff87c0039a0 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c00b980 con 0x7ff87c004d00 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.798+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff87c00bb40 con 0x7ff89015e0a0 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff87c0039a0 0x7ff87c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff8867fc640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff87c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88f7fe640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c004d00 0x7ff87c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 -- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 msgr2=0x7ff87c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff89015e0a0 0x7ff87c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 -- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c004d00 msgr2=0x7ff87c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:54.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c004d00 0x7ff87c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88effd640 1 -- 172.21.15.1:0/3944452004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c003520 con 0x7ff87c0039a0 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.799+0000 7ff88f7fe640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c004d00 0x7ff87c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff88effd640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff87c0039a0 0x7ff87c003d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff880009ac0 tx=0x7ff87c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8a5c8b6d5c7e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88002c600 con 0x7ff87c0039a0 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88002c760 con 0x7ff87c0039a0 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff88e7fc640 1 -- 172.21.15.1:0/3944452004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88002caf0 con 0x7ff87c0039a0 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 -- 172.21.15.1:0/3944452004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff87c0039a0 msgr2=0x7ff87c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 --2- 172.21.15.1:0/3944452004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff87c0039a0 0x7ff87c003d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff880009ac0 tx=0x7ff87c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff88f7fe640 1 -- 172.21.15.1:0/3944452004 reap_dead start 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 -- 172.21.15.1:0/3944452004 shutdown_connections 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 -- 172.21.15.1:0/3944452004 >> 172.21.15.1:0/3944452004 conn(0x7ff8900b2820 msgr2=0x7ff8900b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 -- 172.21.15.1:0/3944452004 shutdown_connections 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.800+0000 7ff895a0b640 1 -- 172.21.15.1:0/3944452004 wait complete. 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.801+0000 7ff895a0b640 1 Processor -- start 2024-03-24T10:34:54.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 -- start start 2024-03-24T10:34:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c0039a0 0x7ff8900c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8900ca300 0x7ff8900cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff8900cc2d0 con 0x7ff8900cbe50 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff8900c6ff0 con 0x7ff87c0039a0 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.802+0000 7ff895a0b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff8900cd900 con 0x7ff8900ca300 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47524/0 (socket says 172.21.15.1:47524) 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 -- 172.21.15.1:0/2835584478 learned_addr learned my addr 172.21.15.1:0/2835584478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff8867fc640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8900ca300 0x7ff8900cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 -- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8900ca300 msgr2=0x7ff8900cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88effd640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c0039a0 0x7ff8900c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8900ca300 0x7ff8900cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 -- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c0039a0 msgr2=0x7ff8900c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c0039a0 0x7ff8900c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:54.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 -- 172.21.15.1:0/2835584478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8900c94a0 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88f7fe640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff8900b35a0 tx=0x7ff88800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88800eb90 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88800ed10 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8900ca830 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888013660 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.803+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8900c6c80 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.804+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8900cad50 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.805+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff8880137c0 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.805+0000 7ff88cff9640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 0x7ff87006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.805+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff88808a5a0 con 0x7ff8900cbe50 2024-03-24T10:34:54.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.805+0000 7ff88effd640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 0x7ff87006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.805+0000 7ff88effd640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 0x7ff87006dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff880036000 tx=0x7ff880030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:54.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.809+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff8900cad50 con 0x7ff8900cbe50 2024-03-24T10:34:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.992+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8900c6ae0 con 0x7ff8900cbe50 2024-03-24T10:34:54.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.993+0000 7ff88cff9640 1 -- 172.21.15.1:0/2835584478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16550 (secure 0 0 0) 0x7ff8900c6ae0 con 0x7ff8900cbe50 2024-03-24T10:34:54.993 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:54.993 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:53.709157+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 msgr2=0x7ff87006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 0x7ff87006dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff880036000 tx=0x7ff880030000 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 msgr2=0x7ff8901f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff8900b35a0 tx=0x7ff88800e530 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 shutdown_connections 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff87006b810 0x7ff87006dcd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8900cbe50 0x7ff8901f70d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8900ca300 0x7ff8900cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 --2- 172.21.15.1:0/2835584478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff87c0039a0 0x7ff8900c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 >> 172.21.15.1:0/2835584478 conn(0x7ff8900b2820 msgr2=0x7ff8900b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 shutdown_connections 2024-03-24T10:34:55.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:54.996+0000 7ff895a0b640 1 -- 172.21.15.1:0/2835584478 wait complete. 2024-03-24T10:34:55.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T10:34:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.111+0000 7f917a288640 1 Processor -- start 2024-03-24T10:34:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.111+0000 7f917a288640 1 -- start start 2024-03-24T10:34:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f917a288640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f917a288640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 0x7f91740d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f917a288640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91740c7a40 con 0x7f91741615f0 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f917a288640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f91740cfd50 con 0x7f9174162610 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f917a288640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f91740b55b0 con 0x7f91740b4e40 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f9178825640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 0x7f91740d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f9178825640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 0x7f91740d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:51528/0 (socket says 172.21.15.1:51528) 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f9173fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f91741615f0 0x7f9174161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49858/0 (socket says 172.21.15.1:49858) 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f9178825640 1 -- 172.21.15.1:0/1654631398 learned_addr learned my addr 172.21.15.1:0/1654631398 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:55.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:34834/0 (socket says 172.21.15.1:34834) 2024-03-24T10:34:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 -- 172.21.15.1:0/1654631398 >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 msgr2=0x7f91740d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 --2- 172.21.15.1:0/1654631398 >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 0x7f91740d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 -- 172.21.15.1:0/1654631398 >> v1:172.21.15.3:6789/0 conn(0x7f91741615f0 legacy=0x7f9174161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.113+0000 7f91737fe640 1 -- 172.21.15.1:0/1654631398 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91740b52d0 con 0x7f91740b4e40 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.114+0000 7f91737fe640 1 --2- 172.21.15.1:0/1654631398 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f91680097a0 tx=0x7f916802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43da516854ef1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.114+0000 7f9172ffd640 1 -- 172.21.15.1:0/1654631398 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168031830 con 0x7f91740b4e40 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.114+0000 7f9172ffd640 1 -- 172.21.15.1:0/1654631398 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9168031990 con 0x7f91740b4e40 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.114+0000 7f9172ffd640 1 -- 172.21.15.1:0/1654631398 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168031d20 con 0x7f91740b4e40 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 -- 172.21.15.1:0/1654631398 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 msgr2=0x7f91740d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 --2- 172.21.15.1:0/1654631398 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f91680097a0 tx=0x7f916802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 -- 172.21.15.1:0/1654631398 shutdown_connections 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 --2- 172.21.15.1:0/1654631398 >> v2:172.21.15.3:3300/0 conn(0x7f9174162610 0x7f91740d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 --2- 172.21.15.1:0/1654631398 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91740b4e40 0x7f91740d4a10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 -- 172.21.15.1:0/1654631398 >> 172.21.15.1:0/1654631398 conn(0x7f91740d3ef0 msgr2=0x7f91740d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 -- 172.21.15.1:0/1654631398 shutdown_connections 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.115+0000 7f917a288640 1 -- 172.21.15.1:0/1654631398 wait complete. 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.116+0000 7f917a288640 1 Processor -- start 2024-03-24T10:34:55.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.116+0000 7f917a288640 1 -- start start 2024-03-24T10:34:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91741615f0 0x7f91741f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 0x7f91740b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f91741f7a10 con 0x7f91741f7e30 2024-03-24T10:34:55.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f91740cdc50 con 0x7f9174162610 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f917a288640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f91741f7350 con 0x7f91741615f0 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f9178825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f9178825640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47540/0 (socket says 172.21.15.1:47540) 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f9178825640 1 -- 172.21.15.1:0/1812912005 learned_addr learned my addr 172.21.15.1:0/1812912005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f91737fe640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 0x7f91740b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.117+0000 7f9173fff640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91741615f0 0x7f91741f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 -- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91741615f0 msgr2=0x7f91741f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91741615f0 0x7f91741f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 -- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 msgr2=0x7f91740b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 0x7f91740b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 -- 172.21.15.1:0/1812912005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9168009400 con 0x7f91741f7e30 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f91737fe640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 0x7f91740b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9178825640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f916c00efc0 tx=0x7f916c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c009b60 con 0x7f91741f7e30 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c00f040 con 0x7f91741f7e30 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91741f7630 con 0x7f91741f7e30 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c003990 con 0x7f91741f7e30 2024-03-24T10:34:55.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.118+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9174156970 con 0x7f91741f7e30 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.119+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91741f71b0 con 0x7f91741f7e30 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.119+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f916c003af0 con 0x7f91741f7e30 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.120+0000 7f9170ff9640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.120+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f916c08a6f0 con 0x7f91741f7e30 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.120+0000 7f9173fff640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:55.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.120+0000 7f9173fff640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 0x7f916006dd00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9164000f60 tx=0x7f9164007c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:55.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.124+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f91741f71b0 con 0x7f91741f7e30 2024-03-24T10:34:55.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:55.411+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f91741f71b0 con 0x7f91741f7e30 2024-03-24T10:34:55.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:34:55.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:34:56.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.360+0000 7f9170ff9640 1 -- 172.21.15.1:0/1812912005 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f91741f71b0 con 0x7f91741f7e30 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.362+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 msgr2=0x7f916006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.362+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 0x7f916006dd00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9164000f60 tx=0x7f9164007c80 comp rx=0 tx=0).stop 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.362+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 msgr2=0x7f917406c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.362+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f916c00efc0 tx=0x7f916c00c490 comp rx=0 tx=0).stop 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 shutdown_connections 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91741f7e30 0x7f917406c030 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9174162610 0x7f91740b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 --2- 172.21.15.1:0/1812912005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f91741615f0 0x7f91741f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 >> 172.21.15.1:0/1812912005 conn(0x7f91740d3ef0 msgr2=0x7f91740d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 shutdown_connections 2024-03-24T10:34:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.363+0000 7f917a288640 1 -- 172.21.15.1:0/1812912005 wait complete. 2024-03-24T10:34:56.374 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T10:34:56.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T10:34:56.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.478+0000 7fbc53cc3640 1 Processor -- start 2024-03-24T10:34:56.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.478+0000 7fbc53cc3640 1 -- start start 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc53cc3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc53cc3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc53cc3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c0c7a40 con 0x7fbc4c0b4e40 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc53cc3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbc4c0cfd50 con 0x7fbc4c1615f0 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc53cc3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c0b55b0 con 0x7fbc4c162610 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc51a5f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc5125e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbc4c0b4e40 0x7fbc4c161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60896/0 (socket says 172.21.15.1:60896) 2024-03-24T10:34:56.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc5125e640 1 -- 172.21.15.1:0/3259851166 learned_addr learned my addr 172.21.15.1:0/3259851166 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.479+0000 7fbc51a5f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47542/0 (socket says 172.21.15.1:47542) 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc52260640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc51a5f640 1 -- 172.21.15.1:0/3259851166 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 msgr2=0x7fbc4c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c0d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc51a5f640 1 -- 172.21.15.1:0/3259851166 >> v1:172.21.15.1:6789/0 conn(0x7fbc4c0b4e40 legacy=0x7fbc4c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc50a5d640 1 -- 172.21.15.1:0/3259851166 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852493422 0 0) 0x7fbc4c0c7a40 con 0x7fbc4c0b4e40 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc51a5f640 1 -- 172.21.15.1:0/3259851166 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0b52d0 con 0x7fbc4c1615f0 2024-03-24T10:34:56.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.480+0000 7fbc52260640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c0d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbc480097a0 tx=0x7fbc4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a0c498e5142b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc50a5d640 1 -- 172.21.15.1:0/3259851166 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48031830 con 0x7fbc4c1615f0 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc50a5d640 1 -- 172.21.15.1:0/3259851166 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc48031990 con 0x7fbc4c1615f0 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc50a5d640 1 -- 172.21.15.1:0/3259851166 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48031d20 con 0x7fbc4c1615f0 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 -- 172.21.15.1:0/3259851166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c1615f0 msgr2=0x7fbc4c0d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbc480097a0 tx=0x7fbc4802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 -- 172.21.15.1:0/3259851166 shutdown_connections 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c0d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/3259851166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0d4a10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 -- 172.21.15.1:0/3259851166 >> 172.21.15.1:0/3259851166 conn(0x7fbc4c0d3ef0 msgr2=0x7fbc4c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:56.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 -- 172.21.15.1:0/3259851166 shutdown_connections 2024-03-24T10:34:56.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.481+0000 7fbc53cc3640 1 -- 172.21.15.1:0/3259851166 wait complete. 2024-03-24T10:34:56.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 Processor -- start 2024-03-24T10:34:56.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 -- start start 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c158d10 con 0x7fbc4c155430 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c157730 con 0x7fbc4c1615f0 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc53cc3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c157a10 con 0x7fbc4c162610 2024-03-24T10:34:56.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc52260640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc52260640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47554/0 (socket says 172.21.15.1:47554) 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.483+0000 7fbc52260640 1 -- 172.21.15.1:0/652937021 learned_addr learned my addr 172.21.15.1:0/652937021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc5125e640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 -- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 msgr2=0x7fbc4c1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 -- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 msgr2=0x7fbc4c0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 -- 172.21.15.1:0/652937021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc48009400 con 0x7fbc4c155430 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc52260640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbc44007e60 tx=0x7fbc4400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44015600 con 0x7fbc4c155430 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4400f040 con 0x7fbc4c155430 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc440159a0 con 0x7fbc4c155430 2024-03-24T10:34:56.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c157cf0 con 0x7fbc4c155430 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.484+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c1582b0 con 0x7fbc4c155430 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.485+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c1584d0 con 0x7fbc4c155430 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.486+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc4401e3b0 con 0x7fbc4c155430 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.486+0000 7fbc427fc640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 0x7fbc3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.486+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fbc4408a470 con 0x7fbc4c155430 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.486+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 0x7fbc3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:56.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.487+0000 7fbc51a5f640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 0x7fbc3006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc48036000 tx=0x7fbc48008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:56.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.491+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc44054f40 con 0x7fbc4c155430 2024-03-24T10:34:56.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:56.716+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbc4c04f470 con 0x7fbc4c155430 2024-03-24T10:34:57.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.367+0000 7fbc427fc640 1 -- 172.21.15.1:0/652937021 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fbc44058600 con 0x7fbc4c155430 2024-03-24T10:34:57.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 msgr2=0x7fbc3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 0x7fbc3006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc48036000 tx=0x7fbc48008030 comp rx=0 tx=0).stop 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 msgr2=0x7fbc4c0cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbc44007e60 tx=0x7fbc4400d4a0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 shutdown_connections 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc3006b810 0x7fbc3006dcd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc4c155430 0x7fbc4c0cb3f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc4c162610 0x7fbc4c1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 --2- 172.21.15.1:0/652937021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc4c1615f0 0x7fbc4c0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 >> 172.21.15.1:0/652937021 conn(0x7fbc4c0d3ef0 msgr2=0x7fbc4c06ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 shutdown_connections 2024-03-24T10:34:57.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.370+0000 7fbc53cc3640 1 -- 172.21.15.1:0/652937021 wait complete. 2024-03-24T10:34:57.381 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T10:34:57.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:57.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.487+0000 7f3429495640 1 Processor -- start 2024-03-24T10:34:57.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.487+0000 7f3429495640 1 -- start start 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3429495640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3429495640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34240d1000 0x7f34240b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3429495640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f342415e770 con 0x7f342415e390 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3429495640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f34240b3190 con 0x7f34240d1000 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3429495640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f34240d4e40 con 0x7f34240d2bb0 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3422ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f342415e390 0x7f34240d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60898/0 (socket says 172.21.15.1:60898) 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f34237fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34240d1000 0x7f34240b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f3422ffd640 1 -- 172.21.15.1:0/2732715231 learned_addr learned my addr 172.21.15.1:0/2732715231 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.488+0000 7f34227fc640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.489+0000 7f34237fe640 1 -- 172.21.15.1:0/2732715231 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 msgr2=0x7f34240b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.489+0000 7f34237fe640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.489+0000 7f34237fe640 1 -- 172.21.15.1:0/2732715231 >> v1:172.21.15.1:6789/0 conn(0x7f342415e390 legacy=0x7f34240d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.489+0000 7f34237fe640 1 -- 172.21.15.1:0/2732715231 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34240d1e40 con 0x7f34240d1000 2024-03-24T10:34:57.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.489+0000 7f3421ffb640 1 -- 172.21.15.1:0/2732715231 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229586860 0 0) 0x7f341c001110 con 0x7f342415e390 2024-03-24T10:34:57.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f34237fe640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240d1000 0x7f34240b40d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3414009d90 tx=0x7f341402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca2b7284ac7872 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3421ffb640 1 -- 172.21.15.1:0/2732715231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341402cb30 con 0x7f34240d1000 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3421ffb640 1 -- 172.21.15.1:0/2732715231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341402cc90 con 0x7f34240d1000 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3421ffb640 1 -- 172.21.15.1:0/2732715231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414034440 con 0x7f34240d1000 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 -- 172.21.15.1:0/2732715231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240d1000 msgr2=0x7f34240b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240d1000 0x7f34240b40d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3414009d90 tx=0x7f341402bd80 comp rx=0 tx=0).stop 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 -- 172.21.15.1:0/2732715231 shutdown_connections 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 --2- 172.21.15.1:0/2732715231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34240d1000 0x7f34240b40d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 -- 172.21.15.1:0/2732715231 >> 172.21.15.1:0/2732715231 conn(0x7f34240d3ec0 msgr2=0x7f34240d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.490+0000 7f3429495640 1 -- 172.21.15.1:0/2732715231 shutdown_connections 2024-03-24T10:34:57.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.491+0000 7f3429495640 1 -- 172.21.15.1:0/2732715231 wait complete. 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.492+0000 7f3429495640 1 Processor -- start 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.492+0000 7f3429495640 1 -- start start 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34240d1000 0x7f3424161570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f34240b8ea0 con 0x7f342415e390 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f34240b9180 con 0x7f34240d1000 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f34240b9460 con 0x7f34240d2bb0 2024-03-24T10:34:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47580/0 (socket says 172.21.15.1:47580) 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 -- 172.21.15.1:0/591354962 learned_addr learned my addr 172.21.15.1:0/591354962 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3422ffd640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34240d1000 0x7f3424161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34227fc640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 -- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 msgr2=0x7f34240ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 -- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34240d1000 msgr2=0x7f3424161570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34240d1000 0x7f3424161570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 -- 172.21.15.1:0/591354962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414009a70 con 0x7f342415e390 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f34237fe640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3414009d90 tx=0x7f3414004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414009450 con 0x7f342415e390 2024-03-24T10:34:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3414034440 con 0x7f342415e390 2024-03-24T10:34:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34240b9740 con 0x7f342415e390 2024-03-24T10:34:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140408a0 con 0x7f342415e390 2024-03-24T10:34:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.493+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34241f3780 con 0x7f342415e390 2024-03-24T10:34:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.494+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34240b9c60 con 0x7f342415e390 2024-03-24T10:34:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.495+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3414040a00 con 0x7f342415e390 2024-03-24T10:34:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.495+0000 7f340b7fe640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 0x7f341006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.495+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f34140accb0 con 0x7f342415e390 2024-03-24T10:34:57.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.495+0000 7f3422ffd640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 0x7f341006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.496+0000 7f3422ffd640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 0x7f341006dcd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f341c007fd0 tx=0x7f341c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.500+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f34240b9c60 con 0x7f342415e390 2024-03-24T10:34:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.777+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34240b9c60 con 0x7f342415e390 2024-03-24T10:34:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.778+0000 7f340b7fe640 1 -- 172.21.15.1:0/591354962 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5163 (secure 0 0 0) 0x7f34240b9c60 con 0x7f342415e390 2024-03-24T10:34:57.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:57.783 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:34:56.717691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:34:57.783 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 msgr2=0x7f341006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 0x7f341006dcd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f341c007fd0 tx=0x7f341c007de0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 msgr2=0x7f34241f6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3414009d90 tx=0x7f3414004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 shutdown_connections 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f341006b810 0x7f341006dcd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f342415e390 0x7f34241f6f50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f34240d2bb0 0x7f34240ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 --2- 172.21.15.1:0/591354962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34240d1000 0x7f3424161570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 >> 172.21.15.1:0/591354962 conn(0x7f34240d3ec0 msgr2=0x7f34240b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 shutdown_connections 2024-03-24T10:34:57.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.780+0000 7f3429495640 1 -- 172.21.15.1:0/591354962 wait complete. 2024-03-24T10:34:57.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:34:57.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.892+0000 7f9ad515e640 1 Processor -- start 2024-03-24T10:34:57.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.892+0000 7f9ad515e640 1 -- start start 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 0x7f9ad00d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ad00d1000 0x7f9ad00d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad00cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad00b3190 con 0x7f9ad00d1000 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad015e770 con 0x7f9ad015e390 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ad515e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad00d4e40 con 0x7f9ad00b3320 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ac659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ad00d1000 0x7f9ad00d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ac659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ad00d1000 0x7f9ad00d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47596/0 (socket says 172.21.15.1:47596) 2024-03-24T10:34:57.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9ac659c640 1 -- 172.21.15.1:0/52920174 learned_addr learned my addr 172.21.15.1:0/52920174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9aced9d640 1 --2- 172.21.15.1:0/52920174 >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 0x7f9ad00d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.893+0000 7f9acf59e640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad00cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 -- 172.21.15.1:0/52920174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 msgr2=0x7f9ad00cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad00cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 -- 172.21.15.1:0/52920174 >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 msgr2=0x7f9ad00d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 --2- 172.21.15.1:0/52920174 >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 0x7f9ad00d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 -- 172.21.15.1:0/52920174 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad00d1e40 con 0x7f9ad00d1000 2024-03-24T10:34:57.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9aced9d640 1 --2- 172.21.15.1:0/52920174 >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 0x7f9ad00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ac659c640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad00d29e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00d3040 tx=0x7f9ab802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994ad5b11afe0298 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.894+0000 7f9ace59c640 1 -- 172.21.15.1:0/52920174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8031750 con 0x7f9ad00d1000 2024-03-24T10:34:57.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ace59c640 1 -- 172.21.15.1:0/52920174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab80318b0 con 0x7f9ad00d1000 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ace59c640 1 -- 172.21.15.1:0/52920174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8031c40 con 0x7f9ad00d1000 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 -- 172.21.15.1:0/52920174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 msgr2=0x7f9ad00d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad00d29e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00d3040 tx=0x7f9ab802b7f0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 -- 172.21.15.1:0/52920174 shutdown_connections 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad00cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 --2- 172.21.15.1:0/52920174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad00d29e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 --2- 172.21.15.1:0/52920174 >> v2:172.21.15.3:3300/0 conn(0x7f9ad015e390 0x7f9ad00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 -- 172.21.15.1:0/52920174 >> 172.21.15.1:0/52920174 conn(0x7f9ad00d3ec0 msgr2=0x7f9ad00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 -- 172.21.15.1:0/52920174 shutdown_connections 2024-03-24T10:34:57.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.895+0000 7f9ad515e640 1 -- 172.21.15.1:0/52920174 wait complete. 2024-03-24T10:34:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.897+0000 7f9ad515e640 1 Processor -- start 2024-03-24T10:34:57.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.897+0000 7f9ad515e640 1 -- start start 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad01599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 0x7f9ad00cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad01f3d80 con 0x7f9ad00d1000 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad015a200 con 0x7f9ad015e390 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ad515e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad015a4e0 con 0x7f9ad00b3320 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ac659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ac659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47612/0 (socket says 172.21.15.1:47612) 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.898+0000 7f9ac659c640 1 -- 172.21.15.1:0/1175559853 learned_addr learned my addr 172.21.15.1:0/1175559853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9aced9d640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad01599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9acf59e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 0x7f9ad00cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 -- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 msgr2=0x7f9ad01599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad01599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 -- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 msgr2=0x7f9ad00cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 0x7f9ad00cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 -- 172.21.15.1:0/1175559853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab80094d0 con 0x7f9ad00d1000 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9acf59e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 0x7f9ad00cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac659c640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9ab802b830 tx=0x7f9ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8033c80 con 0x7f9ad00d1000 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab8033e00 con 0x7f9ad00d1000 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad015a7c0 con 0x7f9ad00d1000 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab803c640 con 0x7f9ad00d1000 2024-03-24T10:34:57.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.899+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad015ad80 con 0x7f9ad00d1000 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.900+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad015a620 con 0x7f9ad00d1000 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.901+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ab803a040 con 0x7f9ad00d1000 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.901+0000 7f9ac7fff640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 0x7f9ab006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.901+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f9ab80abe40 con 0x7f9ad00d1000 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.901+0000 7f9aced9d640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 0x7f9ab006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:57.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.902+0000 7f9aced9d640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 0x7f9ab006dcd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:57.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:57.906+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ad015a620 con 0x7f9ad00d1000 2024-03-24T10:34:58.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.154+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ad015a620 con 0x7f9ad00d1000 2024-03-24T10:34:58.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.155+0000 7f9ac7fff640 1 -- 172.21.15.1:0/1175559853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16537 (secure 0 0 0) 0x7f9ad015a620 con 0x7f9ad00d1000 2024-03-24T10:34:58.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:58.155 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":34,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:34:55.356116+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:34:58.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 msgr2=0x7f9ab006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 0x7f9ab006dcd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0006fd0 tx=0x7f9ac0008040 comp rx=0 tx=0).stop 2024-03-24T10:34:58.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 msgr2=0x7f9ad0150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9ab802b830 tx=0x7f9ab80046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 shutdown_connections 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9ab006b810 0x7f9ab006dcd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad015e390 0x7f9ad00cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ad00d1000 0x7f9ad0150bd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 --2- 172.21.15.1:0/1175559853 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9ad00b3320 0x7f9ad01599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 >> 172.21.15.1:0/1175559853 conn(0x7f9ad00d3ec0 msgr2=0x7f9ad00cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 shutdown_connections 2024-03-24T10:34:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.157+0000 7f9ad515e640 1 -- 172.21.15.1:0/1175559853 wait complete. 2024-03-24T10:34:58.168 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T10:34:58.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:34:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.270+0000 7f22a8920640 1 Processor -- start 2024-03-24T10:34:58.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.270+0000 7f22a8920640 1 -- start start 2024-03-24T10:34:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a8920640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f22a40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a8920640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a415e770 con 0x7f22a40d1000 2024-03-24T10:34:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a8920640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a40d4e40 con 0x7f22a40b4ba0 2024-03-24T10:34:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a8920640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40b3190 con 0x7f22a415e390 2024-03-24T10:34:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f22a40d1000 0x7f22a40cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60908/0 (socket says 172.21.15.1:60908) 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a1d9b640 1 -- 172.21.15.1:0/3992889412 learned_addr learned my addr 172.21.15.1:0/3992889412 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a259c640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f22a40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072483822 0 0) 0x7f228c001110 con 0x7f22a40d1000 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.271+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2290003520 con 0x7f22a40d1000 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240439980 0 0) 0x7f22a40d4e40 con 0x7f22a40b4ba0 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f228c001110 con 0x7f22a40b4ba0 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2169283463 0 0) 0x7f2290003520 con 0x7f22a40d1000 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 msgr2=0x7f22a40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f22a40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 >> v1:172.21.15.3:6789/0 conn(0x7f22a40b4ba0 legacy=0x7f22a40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40d1e40 con 0x7f22a40d1000 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f228c001800 con 0x7f22a40d1000 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 >> v1:172.21.15.1:6789/0 conn(0x7f22a40d1000 legacy=0x7f22a40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2290003b30 0x7f2290003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f2290004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f2290005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f229000bb10 con 0x7f22900047c0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f229000bc80 con 0x7f2290003b30 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f229000be40 con 0x7f22a415e390 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f2290004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a259c640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2290003b30 0x7f2290003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a2d9d640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f2290005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 -- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 msgr2=0x7f2290005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f2290005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 -- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2290003b30 msgr2=0x7f2290003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2290003b30 0x7f2290003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a1d9b640 1 -- 172.21.15.1:0/3992889412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2290003520 con 0x7f22900047c0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.272+0000 7f22a2d9d640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a415e390 0x7f2290005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.273+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f2290004c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f22a40d1440 tx=0x7f22900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47732a1795d5812e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.273+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c005ad0 con 0x7f22900047c0 2024-03-24T10:34:58.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.273+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228c005c50 con 0x7f22900047c0 2024-03-24T10:34:58.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.273+0000 7f22a159a640 1 -- 172.21.15.1:0/3992889412 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c02c7e0 con 0x7f22900047c0 2024-03-24T10:34:58.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 -- 172.21.15.1:0/3992889412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 msgr2=0x7f2290004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 --2- 172.21.15.1:0/3992889412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f2290004c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f22a40d1440 tx=0x7f22900057f0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a2d9d640 1 -- 172.21.15.1:0/3992889412 reap_dead start 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 -- 172.21.15.1:0/3992889412 shutdown_connections 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 -- 172.21.15.1:0/3992889412 >> 172.21.15.1:0/3992889412 conn(0x7f22a40d3ec0 msgr2=0x7f22a40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 -- 172.21.15.1:0/3992889412 shutdown_connections 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.274+0000 7f22a8920640 1 -- 172.21.15.1:0/3992889412 wait complete. 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.275+0000 7f22a8920640 1 Processor -- start 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.275+0000 7f22a8920640 1 -- start start 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 0x7f22a40b9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a40b6b50 0x7f22a40b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22a40b8540 con 0x7f22900047c0 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f22a40b97c0 con 0x7f22a40b8e00 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a8920640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40b78c0 con 0x7f22a40b6b50 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47630/0 (socket says 172.21.15.1:47630) 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 -- 172.21.15.1:0/1285385346 learned_addr learned my addr 172.21.15.1:0/1285385346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:58.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 0x7f22a40b9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a2d9d640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a40b6b50 0x7f22a40b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 -- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a40b6b50 msgr2=0x7f22a40b6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a40b6b50 0x7f22a40b6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 -- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 msgr2=0x7f22a40b9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 0x7f22a40b9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 -- 172.21.15.1:0/1285385346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40b7ba0 con 0x7f22900047c0 2024-03-24T10:34:58.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 0x7f22a40b9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.276+0000 7f22a259c640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22940119a0 tx=0x7f2294011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.277+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229400faf0 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.277+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229400fc50 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.277+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a40b7e80 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.277+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294014440 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.277+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a415ccd0 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.278+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a404f4e0 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.278+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22940145a0 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.278+0000 7f229affd640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 0x7f228406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.278+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f229408aa30 con 0x7f22900047c0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.278+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 0x7f228406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.279+0000 7f22a1d9b640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 0x7f228406dd00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f228c034000 tx=0x7f228c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.283+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22940553d0 con 0x7f22900047c0 2024-03-24T10:34:58.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.464+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f22a40cf880 con 0x7f228406b840 2024-03-24T10:34:58.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.470+0000 7f229affd640 1 -- 172.21.15.1:0/1285385346 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237197 (secure 0 0 0) 0x7f228c001780 con 0x7f228406b840 2024-03-24T10:34:58.473 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:58.475 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 msgr2=0x7f228406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 0x7f228406dd00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f228c034000 tx=0x7f228c007000 comp rx=0 tx=0).stop 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 msgr2=0x7f22a415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22940119a0 tx=0x7f2294011e60 comp rx=0 tx=0).stop 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 shutdown_connections 2024-03-24T10:34:58.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f228406b840 0x7f228406dd00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f22a40b6b50 0x7f22a40b6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22a40b8e00 0x7f22a40b9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 --2- 172.21.15.1:0/1285385346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22900047c0 0x7f22a415c790 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 >> 172.21.15.1:0/1285385346 conn(0x7f22a40d3ec0 msgr2=0x7f22a40d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 shutdown_connections 2024-03-24T10:34:58.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.474+0000 7f22a8920640 1 -- 172.21.15.1:0/1285385346 wait complete. 2024-03-24T10:34:58.489 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-24T10:34:56.529084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236672,"kb_used_data":3784,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738048,"statfs":{"total":773094113280,"available":772851761152,"internally_reserved":0,"allocated":3874816,"data_stored":2192672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.194032"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:53.334953+0000","last_change":"2024-03-24T10:34:53.334953+0000","last_active":"2024-03-24T10:34:53.334953+0000","last_peered":"2024-03-24T10:34:53.334953+0000","last_clean":"2024-03-24T10:34:53.334953+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:53.334953+0000","last_undegraded":"2024-03-24T10:34:53.334953+0000","last_fullsized":"2024-03-24T10:34:53.334953+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T10:34:52.332446+0000","last_change":"2024-03-24T10:34:52.332446+0000","last_active":"2024-03-24T10:34:52.332446+0000","last_peered":"2024-03-24T10:34:52.332446+0000","last_clean":"2024-03-24T10:34:52.332446+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:34:52.332446+0000","last_undegraded":"2024-03-24T10:34:52.332446+0000","last_fullsized":"2024-03-24T10:34:52.332446+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-24T10:34:48.485107+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:34:48.485107+0000","last_peered":"2024-03-24T10:34:48.485107+0000","last_clean":"2024-03-24T10:34:48.485107+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:34:48.485107+0000","last_undegraded":"2024-03-24T10:34:48.485107+0000","last_fullsized":"2024-03-24T10:34:48.485107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":929792,"data_stored":716860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27364,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27356,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31520,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31524,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":335872,"data_stored":126492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":716860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:34:58.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:34:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.630+0000 7fdbcd7fd640 1 Processor -- start 2024-03-24T10:34:58.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.630+0000 7fdbcd7fd640 1 -- start start 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbcd7fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc80b74c0 0x7fdbc80d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbcd7fd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdbc806d2d0 0x7fdbc815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbcd7fd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc80b4c70 con 0x7fdbc806c770 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbcd7fd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdbc80d1340 con 0x7fdbc806d2d0 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbcd7fd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc80b4f50 con 0x7fdbc80b74c0 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdbc806d2d0 0x7fdbc815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc67fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdbc806c770 0x7fdbc806cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60918/0 (socket says 172.21.15.1:60918) 2024-03-24T10:34:58.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc67fc640 1 -- 172.21.15.1:0/288310808 learned_addr learned my addr 172.21.15.1:0/288310808 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc77fe640 1 -- 172.21.15.1:0/288310808 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc80b74c0 msgr2=0x7fdbc80d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc77fe640 1 --2- 172.21.15.1:0/288310808 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc80b74c0 0x7fdbc80d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc77fe640 1 -- 172.21.15.1:0/288310808 >> v1:172.21.15.1:6789/0 conn(0x7fdbc806c770 legacy=0x7fdbc806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc5ffb640 1 -- 172.21.15.1:0/288310808 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802412076 0 0) 0x7fdbc80b4c70 con 0x7fdbc806c770 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.631+0000 7fdbc77fe640 1 -- 172.21.15.1:0/288310808 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc80b5230 con 0x7fdbc806d2d0 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.632+0000 7fdbc77fe640 1 --2- 172.21.15.1:0/288310808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc806d2d0 0x7fdbc815d8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009d90 tx=0x7fdbc002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b113d7a6d7198e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.632+0000 7fdbc5ffb640 1 -- 172.21.15.1:0/288310808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc002cc50 con 0x7fdbc806d2d0 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.632+0000 7fdbc5ffb640 1 -- 172.21.15.1:0/288310808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbc002cdb0 con 0x7fdbc806d2d0 2024-03-24T10:34:58.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.632+0000 7fdbc5ffb640 1 -- 172.21.15.1:0/288310808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0034660 con 0x7fdbc806d2d0 2024-03-24T10:34:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/288310808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc806d2d0 msgr2=0x7fdbc815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/288310808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc806d2d0 0x7fdbc815d8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009d90 tx=0x7fdbc002bea0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/288310808 shutdown_connections 2024-03-24T10:34:58.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/288310808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc806d2d0 0x7fdbc815d8f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/288310808 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc80b74c0 0x7fdbc80d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/288310808 >> 172.21.15.1:0/288310808 conn(0x7fdbc80d1c80 msgr2=0x7fdbc80d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/288310808 shutdown_connections 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.633+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/288310808 wait complete. 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.634+0000 7fdbcd7fd640 1 Processor -- start 2024-03-24T10:34:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.634+0000 7fdbcd7fd640 1 -- start start 2024-03-24T10:34:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbc806c770 0x7fdbc809bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc806d2d0 0x7fdbc80b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc81e6720 con 0x7fdbc80b74c0 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc81e6a00 con 0x7fdbc806c770 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbcd7fd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc81e6ce0 con 0x7fdbc806d2d0 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47646/0 (socket says 172.21.15.1:47646) 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 -- 172.21.15.1:0/3135196678 learned_addr learned my addr 172.21.15.1:0/3135196678 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc67fc640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc806d2d0 0x7fdbc80b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc6ffd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbc806c770 0x7fdbc809bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 -- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc806d2d0 msgr2=0x7fdbc80b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc806d2d0 0x7fdbc80b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 -- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbc806c770 msgr2=0x7fdbc809bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbc806c770 0x7fdbc809bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 -- 172.21.15.1:0/3135196678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc0009a70 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.635+0000 7fdbc77fe640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0037040 tx=0x7fdbc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.636+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0003990 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.636+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbc0003af0 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.636+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc81e6fc0 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.636+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0004020 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.636+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc809c010 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.637+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc804f4e0 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.637+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdbc0004180 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.638+0000 7fdbaf7fe640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 0x7fdbb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.638+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fdbc00ac870 con 0x7fdbc80b74c0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.638+0000 7fdbc6ffd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 0x7fdbb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.638+0000 7fdbc6ffd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 0x7fdbb406dcd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800b810 tx=0x7fdbb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.642+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdbc0077320 con 0x7fdbc80b74c0 2024-03-24T10:34:58.664 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-24T10:34:58.662+0000 7fd6a8dda640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:34:58.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.819+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdbc80d0f10 con 0x7fdbb406b810 2024-03-24T10:34:58.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.825+0000 7fdbaf7fe640 1 -- 172.21.15.1:0/3135196678 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7fdbb800b400 con 0x7fdbb406b810 2024-03-24T10:34:58.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:58.829 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 msgr2=0x7fdbb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 0x7fdbb406dcd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800b810 tx=0x7fdbb8008040 comp rx=0 tx=0).stop 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 msgr2=0x7fdbc81f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0037040 tx=0x7fdbc0004a40 comp rx=0 tx=0).stop 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 shutdown_connections 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbb406b810 0x7fdbb406dcd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbc80b74c0 0x7fdbc81f7570 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdbc806d2d0 0x7fdbc80b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 --2- 172.21.15.1:0/3135196678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbc806c770 0x7fdbc809bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 >> 172.21.15.1:0/3135196678 conn(0x7fdbc80d1c80 msgr2=0x7fdbc80cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 shutdown_connections 2024-03-24T10:34:58.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.828+0000 7fdbcd7fd640 1 -- 172.21.15.1:0/3135196678 wait complete. 2024-03-24T10:34:58.843 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-24T10:34:58.529367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237024,"kb_used_data":4208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737696,"statfs":{"total":773094113280,"available":772851400704,"internally_reserved":0,"allocated":4308992,"data_stored":2386164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000925"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012949+0000","last_change":"2024-03-24T10:34:54.358426+0000","last_active":"2024-03-24T10:34:56.012949+0000","last_peered":"2024-03-24T10:34:56.012949+0000","last_clean":"2024-03-24T10:34:56.012949+0000","last_became_active":"2024-03-24T10:34:54.358171+0000","last_became_peered":"2024-03-24T10:34:54.358171+0000","last_unstale":"2024-03-24T10:34:56.012949+0000","last_undegraded":"2024-03-24T10:34:56.012949+0000","last_fullsized":"2024-03-24T10:34:56.012949+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:20.483284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189005+0000","last_change":"2024-03-24T10:34:54.373364+0000","last_active":"2024-03-24T10:34:56.189005+0000","last_peered":"2024-03-24T10:34:56.189005+0000","last_clean":"2024-03-24T10:34:56.189005+0000","last_became_active":"2024-03-24T10:34:54.372142+0000","last_became_peered":"2024-03-24T10:34:54.372142+0000","last_unstale":"2024-03-24T10:34:56.189005+0000","last_undegraded":"2024-03-24T10:34:56.189005+0000","last_fullsized":"2024-03-24T10:34:56.189005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:48.202930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688247+0000","last_change":"2024-03-24T10:34:54.373247+0000","last_active":"2024-03-24T10:34:55.688247+0000","last_peered":"2024-03-24T10:34:55.688247+0000","last_clean":"2024-03-24T10:34:55.688247+0000","last_became_active":"2024-03-24T10:34:54.372917+0000","last_became_peered":"2024-03-24T10:34:54.372917+0000","last_unstale":"2024-03-24T10:34:55.688247+0000","last_undegraded":"2024-03-24T10:34:55.688247+0000","last_fullsized":"2024-03-24T10:34:55.688247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:11.568935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361918+0000","last_change":"2024-03-24T10:34:54.366790+0000","last_active":"2024-03-24T10:34:55.361918+0000","last_peered":"2024-03-24T10:34:55.361918+0000","last_clean":"2024-03-24T10:34:55.361918+0000","last_became_active":"2024-03-24T10:34:54.366497+0000","last_became_peered":"2024-03-24T10:34:54.366497+0000","last_unstale":"2024-03-24T10:34:55.361918+0000","last_undegraded":"2024-03-24T10:34:55.361918+0000","last_fullsized":"2024-03-24T10:34:55.361918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:10:53.219721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188918+0000","last_change":"2024-03-24T10:34:54.373401+0000","last_active":"2024-03-24T10:34:56.188918+0000","last_peered":"2024-03-24T10:34:56.188918+0000","last_clean":"2024-03-24T10:34:56.188918+0000","last_became_active":"2024-03-24T10:34:54.373038+0000","last_became_peered":"2024-03-24T10:34:54.373038+0000","last_unstale":"2024-03-24T10:34:56.188918+0000","last_undegraded":"2024-03-24T10:34:56.188918+0000","last_fullsized":"2024-03-24T10:34:56.188918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:19.371779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188811+0000","last_change":"2024-03-24T10:34:54.376308+0000","last_active":"2024-03-24T10:34:56.188811+0000","last_peered":"2024-03-24T10:34:56.188811+0000","last_clean":"2024-03-24T10:34:56.188811+0000","last_became_active":"2024-03-24T10:34:54.376025+0000","last_became_peered":"2024-03-24T10:34:54.376025+0000","last_unstale":"2024-03-24T10:34:56.188811+0000","last_undegraded":"2024-03-24T10:34:56.188811+0000","last_fullsized":"2024-03-24T10:34:56.188811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:09.764654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363009+0000","last_change":"2024-03-24T10:34:54.363332+0000","last_active":"2024-03-24T10:34:55.363009+0000","last_peered":"2024-03-24T10:34:55.363009+0000","last_clean":"2024-03-24T10:34:55.363009+0000","last_became_active":"2024-03-24T10:34:54.363061+0000","last_became_peered":"2024-03-24T10:34:54.363061+0000","last_unstale":"2024-03-24T10:34:55.363009+0000","last_undegraded":"2024-03-24T10:34:55.363009+0000","last_fullsized":"2024-03-24T10:34:55.363009+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:28:29.040420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362260+0000","last_change":"2024-03-24T10:34:54.362720+0000","last_active":"2024-03-24T10:34:55.362260+0000","last_peered":"2024-03-24T10:34:55.362260+0000","last_clean":"2024-03-24T10:34:55.362260+0000","last_became_active":"2024-03-24T10:34:54.362385+0000","last_became_peered":"2024-03-24T10:34:54.362385+0000","last_unstale":"2024-03-24T10:34:55.362260+0000","last_undegraded":"2024-03-24T10:34:55.362260+0000","last_fullsized":"2024-03-24T10:34:55.362260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.487335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362969+0000","last_change":"2024-03-24T10:34:54.369211+0000","last_active":"2024-03-24T10:34:55.362969+0000","last_peered":"2024-03-24T10:34:55.362969+0000","last_clean":"2024-03-24T10:34:55.362969+0000","last_became_active":"2024-03-24T10:34:54.368755+0000","last_became_peered":"2024-03-24T10:34:54.368755+0000","last_unstale":"2024-03-24T10:34:55.362969+0000","last_undegraded":"2024-03-24T10:34:55.362969+0000","last_fullsized":"2024-03-24T10:34:55.362969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:25:02.632534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362085+0000","last_change":"2024-03-24T10:34:54.375979+0000","last_active":"2024-03-24T10:34:55.362085+0000","last_peered":"2024-03-24T10:34:55.362085+0000","last_clean":"2024-03-24T10:34:55.362085+0000","last_became_active":"2024-03-24T10:34:54.375531+0000","last_became_peered":"2024-03-24T10:34:54.375531+0000","last_unstale":"2024-03-24T10:34:55.362085+0000","last_undegraded":"2024-03-24T10:34:55.362085+0000","last_fullsized":"2024-03-24T10:34:55.362085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:59.163509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363267+0000","last_change":"2024-03-24T10:34:54.373147+0000","last_active":"2024-03-24T10:34:55.363267+0000","last_peered":"2024-03-24T10:34:55.363267+0000","last_clean":"2024-03-24T10:34:55.363267+0000","last_became_active":"2024-03-24T10:34:54.372860+0000","last_became_peered":"2024-03-24T10:34:54.372860+0000","last_unstale":"2024-03-24T10:34:55.363267+0000","last_undegraded":"2024-03-24T10:34:55.363267+0000","last_fullsized":"2024-03-24T10:34:55.363267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:09.764654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688227+0000","last_change":"2024-03-24T10:34:54.374418+0000","last_active":"2024-03-24T10:34:55.688227+0000","last_peered":"2024-03-24T10:34:55.688227+0000","last_clean":"2024-03-24T10:34:55.688227+0000","last_became_active":"2024-03-24T10:34:54.374023+0000","last_became_peered":"2024-03-24T10:34:54.374023+0000","last_unstale":"2024-03-24T10:34:55.688227+0000","last_undegraded":"2024-03-24T10:34:55.688227+0000","last_fullsized":"2024-03-24T10:34:55.688227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:39:49.665438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189058+0000","last_change":"2024-03-24T10:34:54.365685+0000","last_active":"2024-03-24T10:34:56.189058+0000","last_peered":"2024-03-24T10:34:56.189058+0000","last_clean":"2024-03-24T10:34:56.189058+0000","last_became_active":"2024-03-24T10:34:54.365365+0000","last_became_peered":"2024-03-24T10:34:54.365365+0000","last_unstale":"2024-03-24T10:34:56.189058+0000","last_undegraded":"2024-03-24T10:34:56.189058+0000","last_fullsized":"2024-03-24T10:34:56.189058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:09.633622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188972+0000","last_change":"2024-03-24T10:34:54.368328+0000","last_active":"2024-03-24T10:34:56.188972+0000","last_peered":"2024-03-24T10:34:56.188972+0000","last_clean":"2024-03-24T10:34:56.188972+0000","last_became_active":"2024-03-24T10:34:54.367985+0000","last_became_peered":"2024-03-24T10:34:54.367985+0000","last_unstale":"2024-03-24T10:34:56.188972+0000","last_undegraded":"2024-03-24T10:34:56.188972+0000","last_fullsized":"2024-03-24T10:34:56.188972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:15.886484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688068+0000","last_change":"2024-03-24T10:34:54.358461+0000","last_active":"2024-03-24T10:34:55.688068+0000","last_peered":"2024-03-24T10:34:55.688068+0000","last_clean":"2024-03-24T10:34:55.688068+0000","last_became_active":"2024-03-24T10:34:54.358166+0000","last_became_peered":"2024-03-24T10:34:54.358166+0000","last_unstale":"2024-03-24T10:34:55.688068+0000","last_undegraded":"2024-03-24T10:34:55.688068+0000","last_fullsized":"2024-03-24T10:34:55.688068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:59.163509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362589+0000","last_change":"2024-03-24T10:34:54.370835+0000","last_active":"2024-03-24T10:34:55.362589+0000","last_peered":"2024-03-24T10:34:55.362589+0000","last_clean":"2024-03-24T10:34:55.362589+0000","last_became_active":"2024-03-24T10:34:54.370402+0000","last_became_peered":"2024-03-24T10:34:54.370402+0000","last_unstale":"2024-03-24T10:34:55.362589+0000","last_undegraded":"2024-03-24T10:34:55.362589+0000","last_fullsized":"2024-03-24T10:34:55.362589+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.833867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363301+0000","last_change":"2024-03-24T10:34:54.362850+0000","last_active":"2024-03-24T10:34:55.363301+0000","last_peered":"2024-03-24T10:34:55.363301+0000","last_clean":"2024-03-24T10:34:55.363301+0000","last_became_active":"2024-03-24T10:34:54.362569+0000","last_became_peered":"2024-03-24T10:34:54.362569+0000","last_unstale":"2024-03-24T10:34:55.363301+0000","last_undegraded":"2024-03-24T10:34:55.363301+0000","last_fullsized":"2024-03-24T10:34:55.363301+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:05.693003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189023+0000","last_change":"2024-03-24T10:34:54.373689+0000","last_active":"2024-03-24T10:34:56.189023+0000","last_peered":"2024-03-24T10:34:56.189023+0000","last_clean":"2024-03-24T10:34:56.189023+0000","last_became_active":"2024-03-24T10:34:54.372805+0000","last_became_peered":"2024-03-24T10:34:54.372805+0000","last_unstale":"2024-03-24T10:34:56.189023+0000","last_undegraded":"2024-03-24T10:34:56.189023+0000","last_fullsized":"2024-03-24T10:34:56.189023+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:59.599018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188521+0000","last_change":"2024-03-24T10:34:54.376749+0000","last_active":"2024-03-24T10:34:56.188521+0000","last_peered":"2024-03-24T10:34:56.188521+0000","last_clean":"2024-03-24T10:34:56.188521+0000","last_became_active":"2024-03-24T10:34:54.376385+0000","last_became_peered":"2024-03-24T10:34:54.376385+0000","last_unstale":"2024-03-24T10:34:56.188521+0000","last_undegraded":"2024-03-24T10:34:56.188521+0000","last_fullsized":"2024-03-24T10:34:56.188521+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:28:29.040420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189810+0000","last_change":"2024-03-24T10:34:54.358940+0000","last_active":"2024-03-24T10:34:56.189810+0000","last_peered":"2024-03-24T10:34:56.189810+0000","last_clean":"2024-03-24T10:34:56.189810+0000","last_became_active":"2024-03-24T10:34:54.358630+0000","last_became_peered":"2024-03-24T10:34:54.358630+0000","last_unstale":"2024-03-24T10:34:56.189810+0000","last_undegraded":"2024-03-24T10:34:56.189810+0000","last_fullsized":"2024-03-24T10:34:56.189810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:39.742321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363770+0000","last_change":"2024-03-24T10:34:54.362394+0000","last_active":"2024-03-24T10:34:55.363770+0000","last_peered":"2024-03-24T10:34:55.363770+0000","last_clean":"2024-03-24T10:34:55.363770+0000","last_became_active":"2024-03-24T10:34:54.361526+0000","last_became_peered":"2024-03-24T10:34:54.361526+0000","last_unstale":"2024-03-24T10:34:55.363770+0000","last_undegraded":"2024-03-24T10:34:55.363770+0000","last_fullsized":"2024-03-24T10:34:55.363770+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:30:38.614776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363845+0000","last_change":"2024-03-24T10:34:54.361539+0000","last_active":"2024-03-24T10:34:55.363845+0000","last_peered":"2024-03-24T10:34:55.363845+0000","last_clean":"2024-03-24T10:34:55.363845+0000","last_became_active":"2024-03-24T10:34:54.361011+0000","last_became_peered":"2024-03-24T10:34:54.361011+0000","last_unstale":"2024-03-24T10:34:55.363845+0000","last_undegraded":"2024-03-24T10:34:55.363845+0000","last_fullsized":"2024-03-24T10:34:55.363845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:00:55.043226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188163+0000","last_change":"2024-03-24T10:34:54.367326+0000","last_active":"2024-03-24T10:34:56.188163+0000","last_peered":"2024-03-24T10:34:56.188163+0000","last_clean":"2024-03-24T10:34:56.188163+0000","last_became_active":"2024-03-24T10:34:54.367074+0000","last_became_peered":"2024-03-24T10:34:54.367074+0000","last_unstale":"2024-03-24T10:34:56.188163+0000","last_undegraded":"2024-03-24T10:34:56.188163+0000","last_fullsized":"2024-03-24T10:34:56.188163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:00:55.043226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.687967+0000","last_change":"2024-03-24T10:34:54.358241+0000","last_active":"2024-03-24T10:34:55.687967+0000","last_peered":"2024-03-24T10:34:55.687967+0000","last_clean":"2024-03-24T10:34:55.687967+0000","last_became_active":"2024-03-24T10:34:54.357927+0000","last_became_peered":"2024-03-24T10:34:54.357927+0000","last_unstale":"2024-03-24T10:34:55.687967+0000","last_undegraded":"2024-03-24T10:34:55.687967+0000","last_fullsized":"2024-03-24T10:34:55.687967+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.487335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361662+0000","last_change":"2024-03-24T10:34:54.363749+0000","last_active":"2024-03-24T10:34:55.361662+0000","last_peered":"2024-03-24T10:34:55.361662+0000","last_clean":"2024-03-24T10:34:55.361662+0000","last_became_active":"2024-03-24T10:34:54.363471+0000","last_became_peered":"2024-03-24T10:34:54.363471+0000","last_unstale":"2024-03-24T10:34:55.361662+0000","last_undegraded":"2024-03-24T10:34:55.361662+0000","last_fullsized":"2024-03-24T10:34:55.361662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:47.340275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189150+0000","last_change":"2024-03-24T10:34:54.373092+0000","last_active":"2024-03-24T10:34:56.189150+0000","last_peered":"2024-03-24T10:34:56.189150+0000","last_clean":"2024-03-24T10:34:56.189150+0000","last_became_active":"2024-03-24T10:34:54.372002+0000","last_became_peered":"2024-03-24T10:34:54.372002+0000","last_unstale":"2024-03-24T10:34:56.189150+0000","last_undegraded":"2024-03-24T10:34:56.189150+0000","last_fullsized":"2024-03-24T10:34:56.189150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:23.499589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362430+0000","last_change":"2024-03-24T10:34:54.365357+0000","last_active":"2024-03-24T10:34:55.362430+0000","last_peered":"2024-03-24T10:34:55.362430+0000","last_clean":"2024-03-24T10:34:55.362430+0000","last_became_active":"2024-03-24T10:34:54.364999+0000","last_became_peered":"2024-03-24T10:34:54.364999+0000","last_unstale":"2024-03-24T10:34:55.362430+0000","last_undegraded":"2024-03-24T10:34:55.362430+0000","last_fullsized":"2024-03-24T10:34:55.362430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:09:39.797041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688302+0000","last_change":"2024-03-24T10:34:54.359086+0000","last_active":"2024-03-24T10:34:55.688302+0000","last_peered":"2024-03-24T10:34:55.688302+0000","last_clean":"2024-03-24T10:34:55.688302+0000","last_became_active":"2024-03-24T10:34:54.358812+0000","last_became_peered":"2024-03-24T10:34:54.358812+0000","last_unstale":"2024-03-24T10:34:55.688302+0000","last_undegraded":"2024-03-24T10:34:55.688302+0000","last_fullsized":"2024-03-24T10:34:55.688302+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.833867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363842+0000","last_change":"2024-03-24T10:34:54.365333+0000","last_active":"2024-03-24T10:34:55.363842+0000","last_peered":"2024-03-24T10:34:55.363842+0000","last_clean":"2024-03-24T10:34:55.363842+0000","last_became_active":"2024-03-24T10:34:54.365072+0000","last_became_peered":"2024-03-24T10:34:54.365072+0000","last_unstale":"2024-03-24T10:34:55.363842+0000","last_undegraded":"2024-03-24T10:34:55.363842+0000","last_fullsized":"2024-03-24T10:34:55.363842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:29.871511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362336+0000","last_change":"2024-03-24T10:34:54.360486+0000","last_active":"2024-03-24T10:34:55.362336+0000","last_peered":"2024-03-24T10:34:55.362336+0000","last_clean":"2024-03-24T10:34:55.362336+0000","last_became_active":"2024-03-24T10:34:54.360213+0000","last_became_peered":"2024-03-24T10:34:54.360213+0000","last_unstale":"2024-03-24T10:34:55.362336+0000","last_undegraded":"2024-03-24T10:34:55.362336+0000","last_fullsized":"2024-03-24T10:34:55.362336+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:09:39.797041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012792+0000","last_change":"2024-03-24T10:34:54.361283+0000","last_active":"2024-03-24T10:34:56.012792+0000","last_peered":"2024-03-24T10:34:56.012792+0000","last_clean":"2024-03-24T10:34:56.012792+0000","last_became_active":"2024-03-24T10:34:54.360652+0000","last_became_peered":"2024-03-24T10:34:54.360652+0000","last_unstale":"2024-03-24T10:34:56.012792+0000","last_undegraded":"2024-03-24T10:34:56.012792+0000","last_fullsized":"2024-03-24T10:34:56.012792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:29.871511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362261+0000","last_change":"2024-03-24T10:34:54.364162+0000","last_active":"2024-03-24T10:34:55.362261+0000","last_peered":"2024-03-24T10:34:55.362261+0000","last_clean":"2024-03-24T10:34:55.362261+0000","last_became_active":"2024-03-24T10:34:54.363890+0000","last_became_peered":"2024-03-24T10:34:54.363890+0000","last_unstale":"2024-03-24T10:34:55.362261+0000","last_undegraded":"2024-03-24T10:34:55.362261+0000","last_fullsized":"2024-03-24T10:34:55.362261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:59.163509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188488+0000","last_change":"2024-03-24T10:34:54.360207+0000","last_active":"2024-03-24T10:34:56.188488+0000","last_peered":"2024-03-24T10:34:56.188488+0000","last_clean":"2024-03-24T10:34:56.188488+0000","last_became_active":"2024-03-24T10:34:54.359943+0000","last_became_peered":"2024-03-24T10:34:54.359943+0000","last_unstale":"2024-03-24T10:34:56.188488+0000","last_undegraded":"2024-03-24T10:34:56.188488+0000","last_fullsized":"2024-03-24T10:34:56.188488+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:24:25.032749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.013022+0000","last_change":"2024-03-24T10:34:53.355550+0000","last_active":"2024-03-24T10:34:56.013022+0000","last_peered":"2024-03-24T10:34:56.013022+0000","last_clean":"2024-03-24T10:34:56.013022+0000","last_became_active":"2024-03-24T10:34:53.355235+0000","last_became_peered":"2024-03-24T10:34:53.355235+0000","last_unstale":"2024-03-24T10:34:56.013022+0000","last_undegraded":"2024-03-24T10:34:56.013022+0000","last_fullsized":"2024-03-24T10:34:56.013022+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:55:30.770742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.013003+0000","last_change":"2024-03-24T10:34:54.364943+0000","last_active":"2024-03-24T10:34:56.013003+0000","last_peered":"2024-03-24T10:34:56.013003+0000","last_clean":"2024-03-24T10:34:56.013003+0000","last_became_active":"2024-03-24T10:34:54.364685+0000","last_became_peered":"2024-03-24T10:34:54.364685+0000","last_unstale":"2024-03-24T10:34:56.013003+0000","last_undegraded":"2024-03-24T10:34:56.013003+0000","last_fullsized":"2024-03-24T10:34:56.013003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:24:25.032749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188287+0000","last_change":"2024-03-24T10:34:53.360488+0000","last_active":"2024-03-24T10:34:56.188287+0000","last_peered":"2024-03-24T10:34:56.188287+0000","last_clean":"2024-03-24T10:34:56.188287+0000","last_became_active":"2024-03-24T10:34:53.360229+0000","last_became_peered":"2024-03-24T10:34:53.360229+0000","last_unstale":"2024-03-24T10:34:56.188287+0000","last_undegraded":"2024-03-24T10:34:56.188287+0000","last_fullsized":"2024-03-24T10:34:56.188287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:59.945551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363972+0000","last_change":"2024-03-24T10:34:54.371014+0000","last_active":"2024-03-24T10:34:55.363972+0000","last_peered":"2024-03-24T10:34:55.363972+0000","last_clean":"2024-03-24T10:34:55.363972+0000","last_became_active":"2024-03-24T10:34:54.370654+0000","last_became_peered":"2024-03-24T10:34:54.370654+0000","last_unstale":"2024-03-24T10:34:55.363972+0000","last_undegraded":"2024-03-24T10:34:55.363972+0000","last_fullsized":"2024-03-24T10:34:55.363972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:20.483284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188299+0000","last_change":"2024-03-24T10:34:53.354359+0000","last_active":"2024-03-24T10:34:56.188299+0000","last_peered":"2024-03-24T10:34:56.188299+0000","last_clean":"2024-03-24T10:34:56.188299+0000","last_became_active":"2024-03-24T10:34:53.354026+0000","last_became_peered":"2024-03-24T10:34:53.354026+0000","last_unstale":"2024-03-24T10:34:56.188299+0000","last_undegraded":"2024-03-24T10:34:56.188299+0000","last_fullsized":"2024-03-24T10:34:56.188299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:24:24.030243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362455+0000","last_change":"2024-03-24T10:34:54.361310+0000","last_active":"2024-03-24T10:34:55.362455+0000","last_peered":"2024-03-24T10:34:55.362455+0000","last_clean":"2024-03-24T10:34:55.362455+0000","last_became_active":"2024-03-24T10:34:54.361029+0000","last_became_peered":"2024-03-24T10:34:54.361029+0000","last_unstale":"2024-03-24T10:34:55.362455+0000","last_undegraded":"2024-03-24T10:34:55.362455+0000","last_fullsized":"2024-03-24T10:34:55.362455+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:39:49.665438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189788+0000","last_change":"2024-03-24T10:34:53.362407+0000","last_active":"2024-03-24T10:34:56.189788+0000","last_peered":"2024-03-24T10:34:56.189788+0000","last_clean":"2024-03-24T10:34:56.189788+0000","last_became_active":"2024-03-24T10:34:53.362102+0000","last_became_peered":"2024-03-24T10:34:53.362102+0000","last_unstale":"2024-03-24T10:34:56.189788+0000","last_undegraded":"2024-03-24T10:34:56.189788+0000","last_fullsized":"2024-03-24T10:34:56.189788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:47.493705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362367+0000","last_change":"2024-03-24T10:34:54.365043+0000","last_active":"2024-03-24T10:34:55.362367+0000","last_peered":"2024-03-24T10:34:55.362367+0000","last_clean":"2024-03-24T10:34:55.362367+0000","last_became_active":"2024-03-24T10:34:54.364770+0000","last_became_peered":"2024-03-24T10:34:54.364770+0000","last_unstale":"2024-03-24T10:34:55.362367+0000","last_undegraded":"2024-03-24T10:34:55.362367+0000","last_fullsized":"2024-03-24T10:34:55.362367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:39:49.665438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363904+0000","last_change":"2024-03-24T10:34:53.360293+0000","last_active":"2024-03-24T10:34:55.363904+0000","last_peered":"2024-03-24T10:34:55.363904+0000","last_clean":"2024-03-24T10:34:55.363904+0000","last_became_active":"2024-03-24T10:34:53.360002+0000","last_became_peered":"2024-03-24T10:34:53.360002+0000","last_unstale":"2024-03-24T10:34:55.363904+0000","last_undegraded":"2024-03-24T10:34:55.363904+0000","last_fullsized":"2024-03-24T10:34:55.363904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:11:36.510569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688154+0000","last_change":"2024-03-24T10:34:54.366407+0000","last_active":"2024-03-24T10:34:55.688154+0000","last_peered":"2024-03-24T10:34:55.688154+0000","last_clean":"2024-03-24T10:34:55.688154+0000","last_became_active":"2024-03-24T10:34:54.366089+0000","last_became_peered":"2024-03-24T10:34:54.366089+0000","last_unstale":"2024-03-24T10:34:55.688154+0000","last_undegraded":"2024-03-24T10:34:55.688154+0000","last_fullsized":"2024-03-24T10:34:55.688154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:09:39.797041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363827+0000","last_change":"2024-03-24T10:34:53.360038+0000","last_active":"2024-03-24T10:34:55.363827+0000","last_peered":"2024-03-24T10:34:55.363827+0000","last_clean":"2024-03-24T10:34:55.363827+0000","last_became_active":"2024-03-24T10:34:53.359736+0000","last_became_peered":"2024-03-24T10:34:53.359736+0000","last_unstale":"2024-03-24T10:34:55.363827+0000","last_undegraded":"2024-03-24T10:34:55.363827+0000","last_fullsized":"2024-03-24T10:34:55.363827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:33:38.909786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362756+0000","last_change":"2024-03-24T10:34:54.360000+0000","last_active":"2024-03-24T10:34:55.362756+0000","last_peered":"2024-03-24T10:34:55.362756+0000","last_clean":"2024-03-24T10:34:55.362756+0000","last_became_active":"2024-03-24T10:34:54.359725+0000","last_became_peered":"2024-03-24T10:34:54.359725+0000","last_unstale":"2024-03-24T10:34:55.362756+0000","last_undegraded":"2024-03-24T10:34:55.362756+0000","last_fullsized":"2024-03-24T10:34:55.362756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:58.502324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188804+0000","last_change":"2024-03-24T10:34:53.355017+0000","last_active":"2024-03-24T10:34:56.188804+0000","last_peered":"2024-03-24T10:34:56.188804+0000","last_clean":"2024-03-24T10:34:56.188804+0000","last_became_active":"2024-03-24T10:34:53.354741+0000","last_became_peered":"2024-03-24T10:34:53.354741+0000","last_unstale":"2024-03-24T10:34:56.188804+0000","last_undegraded":"2024-03-24T10:34:56.188804+0000","last_fullsized":"2024-03-24T10:34:56.188804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:19.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361443+0000","last_change":"2024-03-24T10:34:54.359506+0000","last_active":"2024-03-24T10:34:55.361443+0000","last_peered":"2024-03-24T10:34:55.361443+0000","last_clean":"2024-03-24T10:34:55.361443+0000","last_became_active":"2024-03-24T10:34:54.359196+0000","last_became_peered":"2024-03-24T10:34:54.359196+0000","last_unstale":"2024-03-24T10:34:55.361443+0000","last_undegraded":"2024-03-24T10:34:55.361443+0000","last_fullsized":"2024-03-24T10:34:55.361443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:39.766168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362874+0000","last_change":"2024-03-24T10:34:53.357328+0000","last_active":"2024-03-24T10:34:55.362874+0000","last_peered":"2024-03-24T10:34:55.362874+0000","last_clean":"2024-03-24T10:34:55.362874+0000","last_became_active":"2024-03-24T10:34:53.357025+0000","last_became_peered":"2024-03-24T10:34:53.357025+0000","last_unstale":"2024-03-24T10:34:55.362874+0000","last_undegraded":"2024-03-24T10:34:55.362874+0000","last_fullsized":"2024-03-24T10:34:55.362874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:05.785999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688417+0000","last_change":"2024-03-24T10:34:54.375443+0000","last_active":"2024-03-24T10:34:55.688417+0000","last_peered":"2024-03-24T10:34:55.688417+0000","last_clean":"2024-03-24T10:34:55.688417+0000","last_became_active":"2024-03-24T10:34:54.375117+0000","last_became_peered":"2024-03-24T10:34:54.375117+0000","last_unstale":"2024-03-24T10:34:55.688417+0000","last_undegraded":"2024-03-24T10:34:55.688417+0000","last_fullsized":"2024-03-24T10:34:55.688417+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:19.285635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189143+0000","last_change":"2024-03-24T10:34:53.365154+0000","last_active":"2024-03-24T10:34:56.189143+0000","last_peered":"2024-03-24T10:34:56.189143+0000","last_clean":"2024-03-24T10:34:56.189143+0000","last_became_active":"2024-03-24T10:34:53.364879+0000","last_became_peered":"2024-03-24T10:34:53.364879+0000","last_unstale":"2024-03-24T10:34:56.189143+0000","last_undegraded":"2024-03-24T10:34:56.189143+0000","last_fullsized":"2024-03-24T10:34:56.189143+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:56.682163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188939+0000","last_change":"2024-03-24T10:34:54.376934+0000","last_active":"2024-03-24T10:34:56.188939+0000","last_peered":"2024-03-24T10:34:56.188939+0000","last_clean":"2024-03-24T10:34:56.188939+0000","last_became_active":"2024-03-24T10:34:54.376667+0000","last_became_peered":"2024-03-24T10:34:54.376667+0000","last_unstale":"2024-03-24T10:34:56.188939+0000","last_undegraded":"2024-03-24T10:34:56.188939+0000","last_fullsized":"2024-03-24T10:34:56.188939+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:05.693003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188476+0000","last_change":"2024-03-24T10:34:53.360922+0000","last_active":"2024-03-24T10:34:56.188476+0000","last_peered":"2024-03-24T10:34:56.188476+0000","last_clean":"2024-03-24T10:34:56.188476+0000","last_became_active":"2024-03-24T10:34:53.359960+0000","last_became_peered":"2024-03-24T10:34:53.359960+0000","last_unstale":"2024-03-24T10:34:56.188476+0000","last_undegraded":"2024-03-24T10:34:56.188476+0000","last_fullsized":"2024-03-24T10:34:56.188476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:57.499818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188505+0000","last_change":"2024-03-24T10:34:54.375596+0000","last_active":"2024-03-24T10:34:56.188505+0000","last_peered":"2024-03-24T10:34:56.188505+0000","last_clean":"2024-03-24T10:34:56.188505+0000","last_became_active":"2024-03-24T10:34:54.375366+0000","last_became_peered":"2024-03-24T10:34:54.375366+0000","last_unstale":"2024-03-24T10:34:56.188505+0000","last_undegraded":"2024-03-24T10:34:56.188505+0000","last_fullsized":"2024-03-24T10:34:56.188505+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:19.285635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688388+0000","last_change":"2024-03-24T10:34:53.360125+0000","last_active":"2024-03-24T10:34:55.688388+0000","last_peered":"2024-03-24T10:34:55.688388+0000","last_clean":"2024-03-24T10:34:55.688388+0000","last_became_active":"2024-03-24T10:34:53.359827+0000","last_became_peered":"2024-03-24T10:34:53.359827+0000","last_unstale":"2024-03-24T10:34:55.688388+0000","last_undegraded":"2024-03-24T10:34:55.688388+0000","last_fullsized":"2024-03-24T10:34:55.688388+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:24:15.710598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688369+0000","last_change":"2024-03-24T10:34:54.373262+0000","last_active":"2024-03-24T10:34:55.688369+0000","last_peered":"2024-03-24T10:34:55.688369+0000","last_clean":"2024-03-24T10:34:55.688369+0000","last_became_active":"2024-03-24T10:34:54.372956+0000","last_became_peered":"2024-03-24T10:34:54.372956+0000","last_unstale":"2024-03-24T10:34:55.688369+0000","last_undegraded":"2024-03-24T10:34:55.688369+0000","last_fullsized":"2024-03-24T10:34:55.688369+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:05.964128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188436+0000","last_change":"2024-03-24T10:34:53.352169+0000","last_active":"2024-03-24T10:34:56.188436+0000","last_peered":"2024-03-24T10:34:56.188436+0000","last_clean":"2024-03-24T10:34:56.188436+0000","last_became_active":"2024-03-24T10:34:53.351894+0000","last_became_peered":"2024-03-24T10:34:53.351894+0000","last_unstale":"2024-03-24T10:34:56.188436+0000","last_undegraded":"2024-03-24T10:34:56.188436+0000","last_fullsized":"2024-03-24T10:34:56.188436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:34:26.990165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361814+0000","last_change":"2024-03-24T10:34:54.361198+0000","last_active":"2024-03-24T10:34:55.361814+0000","last_peered":"2024-03-24T10:34:55.361814+0000","last_clean":"2024-03-24T10:34:55.361814+0000","last_became_active":"2024-03-24T10:34:54.360871+0000","last_became_peered":"2024-03-24T10:34:54.360871+0000","last_unstale":"2024-03-24T10:34:55.361814+0000","last_undegraded":"2024-03-24T10:34:55.361814+0000","last_fullsized":"2024-03-24T10:34:55.361814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:24:58.179990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363970+0000","last_change":"2024-03-24T10:34:53.354081+0000","last_active":"2024-03-24T10:34:55.363970+0000","last_peered":"2024-03-24T10:34:55.363970+0000","last_clean":"2024-03-24T10:34:55.363970+0000","last_became_active":"2024-03-24T10:34:53.353804+0000","last_became_peered":"2024-03-24T10:34:53.353804+0000","last_unstale":"2024-03-24T10:34:55.363970+0000","last_undegraded":"2024-03-24T10:34:55.363970+0000","last_fullsized":"2024-03-24T10:34:55.363970+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:04.737873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189368+0000","last_change":"2024-03-24T10:34:54.371875+0000","last_active":"2024-03-24T10:34:56.189368+0000","last_peered":"2024-03-24T10:34:56.189368+0000","last_clean":"2024-03-24T10:34:56.189368+0000","last_became_active":"2024-03-24T10:34:54.371335+0000","last_became_peered":"2024-03-24T10:34:54.371335+0000","last_unstale":"2024-03-24T10:34:56.189368+0000","last_undegraded":"2024-03-24T10:34:56.189368+0000","last_fullsized":"2024-03-24T10:34:56.189368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:58.848871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361576+0000","last_change":"2024-03-24T10:34:53.354853+0000","last_active":"2024-03-24T10:34:55.361576+0000","last_peered":"2024-03-24T10:34:55.361576+0000","last_clean":"2024-03-24T10:34:55.361576+0000","last_became_active":"2024-03-24T10:34:53.354575+0000","last_became_peered":"2024-03-24T10:34:53.354575+0000","last_unstale":"2024-03-24T10:34:55.361576+0000","last_undegraded":"2024-03-24T10:34:55.361576+0000","last_fullsized":"2024-03-24T10:34:55.361576+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:59.945551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362449+0000","last_change":"2024-03-24T10:34:54.364950+0000","last_active":"2024-03-24T10:34:55.362449+0000","last_peered":"2024-03-24T10:34:55.362449+0000","last_clean":"2024-03-24T10:34:55.362449+0000","last_became_active":"2024-03-24T10:34:54.364656+0000","last_became_peered":"2024-03-24T10:34:54.364656+0000","last_unstale":"2024-03-24T10:34:55.362449+0000","last_undegraded":"2024-03-24T10:34:55.362449+0000","last_fullsized":"2024-03-24T10:34:55.362449+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:11.568935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188708+0000","last_change":"2024-03-24T10:34:53.360586+0000","last_active":"2024-03-24T10:34:56.188708+0000","last_peered":"2024-03-24T10:34:56.188708+0000","last_clean":"2024-03-24T10:34:56.188708+0000","last_became_active":"2024-03-24T10:34:53.360309+0000","last_became_peered":"2024-03-24T10:34:53.360309+0000","last_unstale":"2024-03-24T10:34:56.188708+0000","last_undegraded":"2024-03-24T10:34:56.188708+0000","last_fullsized":"2024-03-24T10:34:56.188708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:00:54.040720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012901+0000","last_change":"2024-03-24T10:34:54.364933+0000","last_active":"2024-03-24T10:34:56.012901+0000","last_peered":"2024-03-24T10:34:56.012901+0000","last_clean":"2024-03-24T10:34:56.012901+0000","last_became_active":"2024-03-24T10:34:54.364653+0000","last_became_peered":"2024-03-24T10:34:54.364653+0000","last_unstale":"2024-03-24T10:34:56.012901+0000","last_undegraded":"2024-03-24T10:34:56.012901+0000","last_fullsized":"2024-03-24T10:34:56.012901+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:00:55.043226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188412+0000","last_change":"2024-03-24T10:34:53.353511+0000","last_active":"2024-03-24T10:34:56.188412+0000","last_peered":"2024-03-24T10:34:56.188412+0000","last_clean":"2024-03-24T10:34:56.188412+0000","last_became_active":"2024-03-24T10:34:53.353213+0000","last_became_peered":"2024-03-24T10:34:53.353213+0000","last_unstale":"2024-03-24T10:34:56.188412+0000","last_undegraded":"2024-03-24T10:34:56.188412+0000","last_fullsized":"2024-03-24T10:34:56.188412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:28.869005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363147+0000","last_change":"2024-03-24T10:34:54.360423+0000","last_active":"2024-03-24T10:34:55.363147+0000","last_peered":"2024-03-24T10:34:55.363147+0000","last_clean":"2024-03-24T10:34:55.363147+0000","last_became_active":"2024-03-24T10:34:54.360171+0000","last_became_peered":"2024-03-24T10:34:54.360171+0000","last_unstale":"2024-03-24T10:34:55.363147+0000","last_undegraded":"2024-03-24T10:34:55.363147+0000","last_fullsized":"2024-03-24T10:34:55.363147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:24:16.713104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361829+0000","last_change":"2024-03-24T10:34:53.356179+0000","last_active":"2024-03-24T10:34:55.361829+0000","last_peered":"2024-03-24T10:34:55.361829+0000","last_clean":"2024-03-24T10:34:55.361829+0000","last_became_active":"2024-03-24T10:34:53.355921+0000","last_became_peered":"2024-03-24T10:34:53.355921+0000","last_unstale":"2024-03-24T10:34:55.361829+0000","last_undegraded":"2024-03-24T10:34:55.361829+0000","last_fullsized":"2024-03-24T10:34:55.361829+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:02.978388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688453+0000","last_change":"2024-03-24T10:34:54.368129+0000","last_active":"2024-03-24T10:34:55.688453+0000","last_peered":"2024-03-24T10:34:55.688453+0000","last_clean":"2024-03-24T10:34:55.688453+0000","last_became_active":"2024-03-24T10:34:54.367845+0000","last_became_peered":"2024-03-24T10:34:54.367845+0000","last_unstale":"2024-03-24T10:34:55.688453+0000","last_undegraded":"2024-03-24T10:34:55.688453+0000","last_fullsized":"2024-03-24T10:34:55.688453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:28:29.040420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363255+0000","last_change":"2024-03-24T10:34:53.353320+0000","last_active":"2024-03-24T10:34:55.363255+0000","last_peered":"2024-03-24T10:34:55.363255+0000","last_clean":"2024-03-24T10:34:55.363255+0000","last_became_active":"2024-03-24T10:34:53.353018+0000","last_became_peered":"2024-03-24T10:34:53.353018+0000","last_unstale":"2024-03-24T10:34:55.363255+0000","last_undegraded":"2024-03-24T10:34:55.363255+0000","last_fullsized":"2024-03-24T10:34:55.363255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:54:00.049166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362046+0000","last_change":"2024-03-24T10:34:54.365533+0000","last_active":"2024-03-24T10:34:55.362046+0000","last_peered":"2024-03-24T10:34:55.362046+0000","last_clean":"2024-03-24T10:34:55.362046+0000","last_became_active":"2024-03-24T10:34:54.364626+0000","last_became_peered":"2024-03-24T10:34:54.364626+0000","last_unstale":"2024-03-24T10:34:55.362046+0000","last_undegraded":"2024-03-24T10:34:55.362046+0000","last_fullsized":"2024-03-24T10:34:55.362046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:05.487335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363602+0000","last_change":"2024-03-24T10:34:53.358869+0000","last_active":"2024-03-24T10:34:55.363602+0000","last_peered":"2024-03-24T10:34:55.363602+0000","last_clean":"2024-03-24T10:34:55.363602+0000","last_became_active":"2024-03-24T10:34:53.358527+0000","last_became_peered":"2024-03-24T10:34:53.358527+0000","last_unstale":"2024-03-24T10:34:55.363602+0000","last_undegraded":"2024-03-24T10:34:55.363602+0000","last_fullsized":"2024-03-24T10:34:55.363602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:08.631116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188612+0000","last_change":"2024-03-24T10:34:54.362080+0000","last_active":"2024-03-24T10:34:56.188612+0000","last_peered":"2024-03-24T10:34:56.188612+0000","last_clean":"2024-03-24T10:34:56.188612+0000","last_became_active":"2024-03-24T10:34:54.361763+0000","last_became_peered":"2024-03-24T10:34:54.361763+0000","last_unstale":"2024-03-24T10:34:56.188612+0000","last_undegraded":"2024-03-24T10:34:56.188612+0000","last_fullsized":"2024-03-24T10:34:56.188612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:11:37.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363974+0000","last_change":"2024-03-24T10:34:53.360975+0000","last_active":"2024-03-24T10:34:55.363974+0000","last_peered":"2024-03-24T10:34:55.363974+0000","last_clean":"2024-03-24T10:34:55.363974+0000","last_became_active":"2024-03-24T10:34:53.360687+0000","last_became_peered":"2024-03-24T10:34:53.360687+0000","last_unstale":"2024-03-24T10:34:55.363974+0000","last_undegraded":"2024-03-24T10:34:55.363974+0000","last_fullsized":"2024-03-24T10:34:55.363974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:18.369273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189373+0000","last_change":"2024-03-24T10:34:54.366764+0000","last_active":"2024-03-24T10:34:56.189373+0000","last_peered":"2024-03-24T10:34:56.189373+0000","last_clean":"2024-03-24T10:34:56.189373+0000","last_became_active":"2024-03-24T10:34:54.365670+0000","last_became_peered":"2024-03-24T10:34:54.365670+0000","last_unstale":"2024-03-24T10:34:56.189373+0000","last_undegraded":"2024-03-24T10:34:56.189373+0000","last_fullsized":"2024-03-24T10:34:56.189373+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:16.541506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188988+0000","last_change":"2024-03-24T10:34:53.363184+0000","last_active":"2024-03-24T10:34:56.188988+0000","last_peered":"2024-03-24T10:34:56.188988+0000","last_clean":"2024-03-24T10:34:56.188988+0000","last_became_active":"2024-03-24T10:34:53.362897+0000","last_became_peered":"2024-03-24T10:34:53.362897+0000","last_unstale":"2024-03-24T10:34:56.188988+0000","last_undegraded":"2024-03-24T10:34:56.188988+0000","last_fullsized":"2024-03-24T10:34:56.188988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:55:30.770742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189196+0000","last_change":"2024-03-24T10:34:54.353640+0000","last_active":"2024-03-24T10:34:56.189196+0000","last_peered":"2024-03-24T10:34:56.189196+0000","last_clean":"2024-03-24T10:34:56.189196+0000","last_became_active":"2024-03-24T10:34:54.353302+0000","last_became_peered":"2024-03-24T10:34:54.353302+0000","last_unstale":"2024-03-24T10:34:56.189196+0000","last_undegraded":"2024-03-24T10:34:56.189196+0000","last_fullsized":"2024-03-24T10:34:56.189196+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:54:01.051673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012766+0000","last_change":"2024-03-24T10:34:53.356951+0000","last_active":"2024-03-24T10:34:56.012766+0000","last_peered":"2024-03-24T10:34:56.012766+0000","last_clean":"2024-03-24T10:34:56.012766+0000","last_became_active":"2024-03-24T10:34:53.356624+0000","last_became_peered":"2024-03-24T10:34:53.356624+0000","last_unstale":"2024-03-24T10:34:56.012766+0000","last_undegraded":"2024-03-24T10:34:56.012766+0000","last_fullsized":"2024-03-24T10:34:56.012766+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:04:46.238462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189643+0000","last_change":"2024-03-24T10:34:54.354156+0000","last_active":"2024-03-24T10:34:56.189643+0000","last_peered":"2024-03-24T10:34:56.189643+0000","last_clean":"2024-03-24T10:34:56.189643+0000","last_became_active":"2024-03-24T10:34:54.353780+0000","last_became_peered":"2024-03-24T10:34:54.353780+0000","last_unstale":"2024-03-24T10:34:56.189643+0000","last_undegraded":"2024-03-24T10:34:56.189643+0000","last_fullsized":"2024-03-24T10:34:56.189643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:57:57.135499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363602+0000","last_change":"2024-03-24T10:34:53.356385+0000","last_active":"2024-03-24T10:34:55.363602+0000","last_peered":"2024-03-24T10:34:55.363602+0000","last_clean":"2024-03-24T10:34:55.363602+0000","last_became_active":"2024-03-24T10:34:53.356122+0000","last_became_peered":"2024-03-24T10:34:53.356122+0000","last_unstale":"2024-03-24T10:34:55.363602+0000","last_undegraded":"2024-03-24T10:34:55.363602+0000","last_fullsized":"2024-03-24T10:34:55.363602+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:10.479487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362818+0000","last_change":"2024-03-24T10:34:54.362827+0000","last_active":"2024-03-24T10:34:55.362818+0000","last_peered":"2024-03-24T10:34:55.362818+0000","last_clean":"2024-03-24T10:34:55.362818+0000","last_became_active":"2024-03-24T10:34:54.362524+0000","last_became_peered":"2024-03-24T10:34:54.362524+0000","last_unstale":"2024-03-24T10:34:55.362818+0000","last_undegraded":"2024-03-24T10:34:55.362818+0000","last_fullsized":"2024-03-24T10:34:55.362818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:05.964128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.687899+0000","last_change":"2024-03-24T10:34:53.365246+0000","last_active":"2024-03-24T10:34:55.687899+0000","last_peered":"2024-03-24T10:34:55.687899+0000","last_clean":"2024-03-24T10:34:55.687899+0000","last_became_active":"2024-03-24T10:34:53.364958+0000","last_became_peered":"2024-03-24T10:34:53.364958+0000","last_unstale":"2024-03-24T10:34:55.687899+0000","last_undegraded":"2024-03-24T10:34:55.687899+0000","last_fullsized":"2024-03-24T10:34:55.687899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:25:01.630028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188765+0000","last_change":"2024-03-24T10:34:54.358222+0000","last_active":"2024-03-24T10:34:56.188765+0000","last_peered":"2024-03-24T10:34:56.188765+0000","last_clean":"2024-03-24T10:34:56.188765+0000","last_became_active":"2024-03-24T10:34:54.357918+0000","last_became_peered":"2024-03-24T10:34:54.357918+0000","last_unstale":"2024-03-24T10:34:56.188765+0000","last_undegraded":"2024-03-24T10:34:56.188765+0000","last_fullsized":"2024-03-24T10:34:56.188765+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:05.740379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361867+0000","last_change":"2024-03-24T10:34:53.354593+0000","last_active":"2024-03-24T10:34:55.361867+0000","last_peered":"2024-03-24T10:34:55.361867+0000","last_clean":"2024-03-24T10:34:55.361867+0000","last_became_active":"2024-03-24T10:34:53.354242+0000","last_became_peered":"2024-03-24T10:34:53.354242+0000","last_unstale":"2024-03-24T10:34:55.361867+0000","last_undegraded":"2024-03-24T10:34:55.361867+0000","last_fullsized":"2024-03-24T10:34:55.361867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:57.499818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.688563+0000","last_change":"2024-03-24T10:34:54.373025+0000","last_active":"2024-03-24T10:34:55.688563+0000","last_peered":"2024-03-24T10:34:55.688563+0000","last_clean":"2024-03-24T10:34:55.688563+0000","last_became_active":"2024-03-24T10:34:54.372675+0000","last_became_peered":"2024-03-24T10:34:54.372675+0000","last_unstale":"2024-03-24T10:34:55.688563+0000","last_undegraded":"2024-03-24T10:34:55.688563+0000","last_fullsized":"2024-03-24T10:34:55.688563+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:09.764654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188385+0000","last_change":"2024-03-24T10:34:53.362861+0000","last_active":"2024-03-24T10:34:56.188385+0000","last_peered":"2024-03-24T10:34:56.188385+0000","last_clean":"2024-03-24T10:34:56.188385+0000","last_became_active":"2024-03-24T10:34:53.362590+0000","last_became_peered":"2024-03-24T10:34:53.362590+0000","last_unstale":"2024-03-24T10:34:56.188385+0000","last_undegraded":"2024-03-24T10:34:56.188385+0000","last_fullsized":"2024-03-24T10:34:56.188385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:02.978388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188191+0000","last_change":"2024-03-24T10:34:54.366786+0000","last_active":"2024-03-24T10:34:56.188191+0000","last_peered":"2024-03-24T10:34:56.188191+0000","last_clean":"2024-03-24T10:34:56.188191+0000","last_became_active":"2024-03-24T10:34:54.366511+0000","last_became_peered":"2024-03-24T10:34:54.366511+0000","last_unstale":"2024-03-24T10:34:56.188191+0000","last_undegraded":"2024-03-24T10:34:56.188191+0000","last_fullsized":"2024-03-24T10:34:56.188191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:20.483284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362131+0000","last_change":"2024-03-24T10:34:53.358654+0000","last_active":"2024-03-24T10:34:55.362131+0000","last_peered":"2024-03-24T10:34:55.362131+0000","last_clean":"2024-03-24T10:34:55.362131+0000","last_became_active":"2024-03-24T10:34:53.358355+0000","last_became_peered":"2024-03-24T10:34:53.358355+0000","last_unstale":"2024-03-24T10:34:55.362131+0000","last_undegraded":"2024-03-24T10:34:55.362131+0000","last_fullsized":"2024-03-24T10:34:55.362131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:24:15.710598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012747+0000","last_change":"2024-03-24T10:34:54.359357+0000","last_active":"2024-03-24T10:34:56.012747+0000","last_peered":"2024-03-24T10:34:56.012747+0000","last_clean":"2024-03-24T10:34:56.012747+0000","last_became_active":"2024-03-24T10:34:54.359092+0000","last_became_peered":"2024-03-24T10:34:54.359092+0000","last_unstale":"2024-03-24T10:34:56.012747+0000","last_undegraded":"2024-03-24T10:34:56.012747+0000","last_fullsized":"2024-03-24T10:34:56.012747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:30:38.614776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.189146+0000","last_change":"2024-03-24T10:34:53.364250+0000","last_active":"2024-03-24T10:34:56.189146+0000","last_peered":"2024-03-24T10:34:56.189146+0000","last_clean":"2024-03-24T10:34:56.189146+0000","last_became_active":"2024-03-24T10:34:53.363899+0000","last_became_peered":"2024-03-24T10:34:53.363899+0000","last_unstale":"2024-03-24T10:34:56.189146+0000","last_undegraded":"2024-03-24T10:34:56.189146+0000","last_fullsized":"2024-03-24T10:34:56.189146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:04:46.238462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188870+0000","last_change":"2024-03-24T10:34:54.358589+0000","last_active":"2024-03-24T10:34:56.188870+0000","last_peered":"2024-03-24T10:34:56.188870+0000","last_clean":"2024-03-24T10:34:56.188870+0000","last_became_active":"2024-03-24T10:34:54.358337+0000","last_became_peered":"2024-03-24T10:34:54.358337+0000","last_unstale":"2024-03-24T10:34:56.188870+0000","last_undegraded":"2024-03-24T10:34:56.188870+0000","last_fullsized":"2024-03-24T10:34:56.188870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:57.684669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.361671+0000","last_change":"2024-03-24T10:34:53.355290+0000","last_active":"2024-03-24T10:34:55.361671+0000","last_peered":"2024-03-24T10:34:55.361671+0000","last_clean":"2024-03-24T10:34:55.361671+0000","last_became_active":"2024-03-24T10:34:53.354330+0000","last_became_peered":"2024-03-24T10:34:53.354330+0000","last_unstale":"2024-03-24T10:34:55.361671+0000","last_undegraded":"2024-03-24T10:34:55.361671+0000","last_fullsized":"2024-03-24T10:34:55.361671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:25:01.630028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188223+0000","last_change":"2024-03-24T10:34:54.365108+0000","last_active":"2024-03-24T10:34:56.188223+0000","last_peered":"2024-03-24T10:34:56.188223+0000","last_clean":"2024-03-24T10:34:56.188223+0000","last_became_active":"2024-03-24T10:34:54.364825+0000","last_became_peered":"2024-03-24T10:34:54.364825+0000","last_unstale":"2024-03-24T10:34:56.188223+0000","last_undegraded":"2024-03-24T10:34:56.188223+0000","last_fullsized":"2024-03-24T10:34:56.188223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:05.964128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363739+0000","last_change":"2024-03-24T10:34:53.355657+0000","last_active":"2024-03-24T10:34:55.363739+0000","last_peered":"2024-03-24T10:34:55.363739+0000","last_clean":"2024-03-24T10:34:55.363739+0000","last_became_active":"2024-03-24T10:34:53.355385+0000","last_became_peered":"2024-03-24T10:34:53.355385+0000","last_unstale":"2024-03-24T10:34:55.363739+0000","last_undegraded":"2024-03-24T10:34:55.363739+0000","last_fullsized":"2024-03-24T10:34:55.363739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:06:53.315986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.363687+0000","last_change":"2024-03-24T10:34:54.370948+0000","last_active":"2024-03-24T10:34:55.363687+0000","last_peered":"2024-03-24T10:34:55.363687+0000","last_clean":"2024-03-24T10:34:55.363687+0000","last_became_active":"2024-03-24T10:34:54.370632+0000","last_became_peered":"2024-03-24T10:34:54.370632+0000","last_unstale":"2024-03-24T10:34:55.363687+0000","last_undegraded":"2024-03-24T10:34:55.363687+0000","last_fullsized":"2024-03-24T10:34:55.363687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:41.203087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.188319+0000","last_change":"2024-03-24T10:34:53.365228+0000","last_active":"2024-03-24T10:34:56.188319+0000","last_peered":"2024-03-24T10:34:56.188319+0000","last_clean":"2024-03-24T10:34:56.188319+0000","last_became_active":"2024-03-24T10:34:53.364949+0000","last_became_peered":"2024-03-24T10:34:53.364949+0000","last_unstale":"2024-03-24T10:34:56.188319+0000","last_undegraded":"2024-03-24T10:34:56.188319+0000","last_fullsized":"2024-03-24T10:34:56.188319+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:30:37.612270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362948+0000","last_change":"2024-03-24T10:34:54.359643+0000","last_active":"2024-03-24T10:34:55.362948+0000","last_peered":"2024-03-24T10:34:55.362948+0000","last_clean":"2024-03-24T10:34:55.362948+0000","last_became_active":"2024-03-24T10:34:54.359373+0000","last_became_peered":"2024-03-24T10:34:54.359373+0000","last_unstale":"2024-03-24T10:34:55.362948+0000","last_undegraded":"2024-03-24T10:34:55.362948+0000","last_fullsized":"2024-03-24T10:34:55.362948+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:53.334953+0000","last_clean_scrub_stamp":"2024-03-24T10:34:53.334953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:19.285635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:56.012863+0000","last_change":"2024-03-24T10:34:53.359910+0000","last_active":"2024-03-24T10:34:56.012863+0000","last_peered":"2024-03-24T10:34:56.012863+0000","last_clean":"2024-03-24T10:34:56.012863+0000","last_became_active":"2024-03-24T10:34:53.359589+0000","last_became_peered":"2024-03-24T10:34:53.359589+0000","last_unstale":"2024-03-24T10:34:56.012863+0000","last_undegraded":"2024-03-24T10:34:56.012863+0000","last_fullsized":"2024-03-24T10:34:56.012863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:34:52.332446+0000","last_clean_scrub_stamp":"2024-03-24T10:34:52.332446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:45:39.178531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":193,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-24T10:34:55.362512+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:34:55.362512+0000","last_peered":"2024-03-24T10:34:55.362512+0000","last_clean":"2024-03-24T10:34:55.362512+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:34:55.362512+0000","last_undegraded":"2024-03-24T10:34:55.362512+0000","last_fullsized":"2024-03-24T10:34:55.362512+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":742109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":145967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":151755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":153285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":151721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":153285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":742019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:34:58.848 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T10:34:58.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T10:34:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.996+0000 7f40dba9d640 1 Processor -- start 2024-03-24T10:34:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.996+0000 7f40dba9d640 1 -- start start 2024-03-24T10:34:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.996+0000 7f40dba9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40d4154be0 0x7f40d406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.996+0000 7f40dba9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d40cf450 0x7f40d40b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40dba9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40d40b40c0 0x7f40d415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40dba9d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f40d4070bd0 con 0x7f40d4154be0 2024-03-24T10:34:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40dba9d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f40d4154fc0 con 0x7f40d40b40c0 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40dba9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f40d4071370 con 0x7f40d40cf450 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40d4154be0 0x7f40d406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40d4154be0 0x7f40d406d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47648/0 (socket says 172.21.15.1:47648) 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 -- 172.21.15.1:0/2777180590 learned_addr learned my addr 172.21.15.1:0/2777180590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9038640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d40cf450 0x7f40d40b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40da03a640 1 --2- 172.21.15.1:0/2777180590 >> v2:172.21.15.3:3300/0 conn(0x7f40d40b40c0 0x7f40d415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 -- 172.21.15.1:0/2777180590 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d40cf450 msgr2=0x7f40d40b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d40cf450 0x7f40d40b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 -- 172.21.15.1:0/2777180590 >> v2:172.21.15.3:3300/0 conn(0x7f40d40b40c0 msgr2=0x7f40d415aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 --2- 172.21.15.1:0/2777180590 >> v2:172.21.15.3:3300/0 conn(0x7f40d40b40c0 0x7f40d415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.997+0000 7f40d9839640 1 -- 172.21.15.1:0/2777180590 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d40b36a0 con 0x7f40d4154be0 2024-03-24T10:34:58.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40d9839640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d4154be0 0x7f40d406d4a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009870 tx=0x7f40d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d65b98c0d88cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:58.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40d8837640 1 -- 172.21.15.1:0/2777180590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00317c0 con 0x7f40d4154be0 2024-03-24T10:34:58.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40d8837640 1 -- 172.21.15.1:0/2777180590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d0031920 con 0x7f40d4154be0 2024-03-24T10:34:58.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40d8837640 1 -- 172.21.15.1:0/2777180590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0031cb0 con 0x7f40d4154be0 2024-03-24T10:34:58.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 -- 172.21.15.1:0/2777180590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d4154be0 msgr2=0x7f40d406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d4154be0 0x7f40d406d4a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009870 tx=0x7f40d002b860 comp rx=0 tx=0).stop 2024-03-24T10:34:59.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 -- 172.21.15.1:0/2777180590 shutdown_connections 2024-03-24T10:34:59.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2777180590 >> v2:172.21.15.3:3300/0 conn(0x7f40d40b40c0 0x7f40d415aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d40cf450 0x7f40d40b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2777180590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d4154be0 0x7f40d406d4a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.998+0000 7f40dba9d640 1 -- 172.21.15.1:0/2777180590 >> 172.21.15.1:0/2777180590 conn(0x7f40d40b1710 msgr2=0x7f40d40b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.999+0000 7f40dba9d640 1 -- 172.21.15.1:0/2777180590 shutdown_connections 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:58.999+0000 7f40dba9d640 1 -- 172.21.15.1:0/2777180590 wait complete. 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 Processor -- start 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 -- start start 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40d40cf450 0x7f40d4155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d4154be0 0x7f40d4156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f40d40cda40 con 0x7f40d40b40c0 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f40d40adb60 con 0x7f40d40cf450 2024-03-24T10:34:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.000+0000 7f40dba9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f40d4149860 con 0x7f40d4154be0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47662/0 (socket says 172.21.15.1:47662) 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 -- 172.21.15.1:0/2125003423 learned_addr learned my addr 172.21.15.1:0/2125003423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9038640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40d40cf450 0x7f40d4155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40da03a640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d4154be0 0x7f40d4156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 -- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d4154be0 msgr2=0x7f40d4156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d4154be0 0x7f40d4156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 -- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40d40cf450 msgr2=0x7f40d4155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40d40cf450 0x7f40d4155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 -- 172.21.15.1:0/2125003423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d00094d0 con 0x7f40d40b40c0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40d9839640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f40d002b8a0 tx=0x7f40d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00329a0 con 0x7f40d40b40c0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d0032b40 con 0x7f40d40b40c0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d40cce30 con 0x7f40d40b40c0 2024-03-24T10:34:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00453f0 con 0x7f40d40b40c0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.001+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d40cd3f0 con 0x7f40d40b40c0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.002+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d404f4e0 con 0x7f40d40b40c0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.003+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f40d004e450 con 0x7f40d40b40c0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.003+0000 7f40c67fc640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 0x7f40b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.003+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f40d00ac500 con 0x7f40d40b40c0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.003+0000 7f40d9038640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 0x7f40b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.004+0000 7f40d9038640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 0x7f40b806dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f40d40af580 tx=0x7f40c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:59.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.011+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f40d0076f20 con 0x7f40d40b40c0 2024-03-24T10:34:59.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.264+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f40d40b2e00 con 0x7f40d40b40c0 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.266+0000 7f40c67fc640 1 -- 172.21.15.1:0/2125003423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f40d007a5e0 con 0x7f40d40b40c0 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:34:59.266 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T10:34:59.267 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 msgr2=0x7f40b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 0x7f40b806dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f40d40af580 tx=0x7f40c8008040 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 msgr2=0x7f40d4149320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f40d002b8a0 tx=0x7f40d00046c0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 shutdown_connections 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f40b806b810 0x7f40b806dcd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f40d4154be0 0x7f40d4156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40d40cf450 0x7f40d4155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 --2- 172.21.15.1:0/2125003423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40d40b40c0 0x7f40d4149320 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 >> 172.21.15.1:0/2125003423 conn(0x7f40d40b1710 msgr2=0x7f40d415d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 shutdown_connections 2024-03-24T10:34:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.268+0000 7f40dba9d640 1 -- 172.21.15.1:0/2125003423 wait complete. 2024-03-24T10:34:59.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:34:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.393+0000 7f1f64ad4640 1 Processor -- start 2024-03-24T10:34:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.393+0000 7f1f64ad4640 1 -- start start 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f64ad4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 0x7f1f600d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f64ad4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f600d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f64ad4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6015e770 con 0x7f1f600d1000 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f64ad4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f1f600b3190 con 0x7f1f6015e390 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f64ad4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1f600d4e40 con 0x7f1f600b4ba0 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f5ed9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f600d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f5e59c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 0x7f1f600d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f5dd9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1f600d1000 0x7f1f600cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:49886/0 (socket says 172.21.15.1:49886) 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.394+0000 7f1f5dd9b640 1 -- 172.21.15.1:0/911367704 learned_addr learned my addr 172.21.15.1:0/911367704 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5d59a640 1 -- 172.21.15.1:0/911367704 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268113047 0 0) 0x7f1f48001110 con 0x7f1f600d1000 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5d59a640 1 -- 172.21.15.1:0/911367704 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f4c003520 con 0x7f1f600d1000 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5ed9d640 1 -- 172.21.15.1:0/911367704 >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 msgr2=0x7f1f600d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5ed9d640 1 --2- 172.21.15.1:0/911367704 >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 0x7f1f600d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5ed9d640 1 -- 172.21.15.1:0/911367704 >> v1:172.21.15.3:6789/0 conn(0x7f1f600d1000 legacy=0x7f1f600cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5ed9d640 1 -- 172.21.15.1:0/911367704 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f600d1e40 con 0x7f1f600b4ba0 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.395+0000 7f1f5e59c640 1 --2- 172.21.15.1:0/911367704 >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 0x7f1f600d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f5ed9d640 1 --2- 172.21.15.1:0/911367704 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f600d29e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f580097a0 tx=0x7f1f5802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1abe68312a9126d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f5d59a640 1 -- 172.21.15.1:0/911367704 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5802f050 con 0x7f1f600b4ba0 2024-03-24T10:34:59.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f5d59a640 1 -- 172.21.15.1:0/911367704 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f58031700 con 0x7f1f600b4ba0 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f5d59a640 1 -- 172.21.15.1:0/911367704 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58031a90 con 0x7f1f600b4ba0 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 -- 172.21.15.1:0/911367704 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 msgr2=0x7f1f600d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/911367704 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f600d29e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f580097a0 tx=0x7f1f5802b7a0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 -- 172.21.15.1:0/911367704 shutdown_connections 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/911367704 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f600d29e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/911367704 >> v2:172.21.15.3:3300/0 conn(0x7f1f6015e390 0x7f1f600d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 -- 172.21.15.1:0/911367704 >> 172.21.15.1:0/911367704 conn(0x7f1f600d3ec0 msgr2=0x7f1f600d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 -- 172.21.15.1:0/911367704 shutdown_connections 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.396+0000 7f1f64ad4640 1 -- 172.21.15.1:0/911367704 wait complete. 2024-03-24T10:34:59.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 Processor -- start 2024-03-24T10:34:59.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 -- start start 2024-03-24T10:34:59.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f601562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 0x7f1f6009f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1f600a1b30 con 0x7f1f600d1000 2024-03-24T10:34:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6009f470 con 0x7f1f6015e390 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.398+0000 7f1f64ad4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6009e860 con 0x7f1f600b4ba0 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47678/0 (socket says 172.21.15.1:47678) 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 -- 172.21.15.1:0/231621214 learned_addr learned my addr 172.21.15.1:0/231621214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5e59c640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f601562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5ed9d640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 0x7f1f6009f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 -- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 msgr2=0x7f1f601562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f601562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 -- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 msgr2=0x7f1f6009f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 0x7f1f6009f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 -- 172.21.15.1:0/231621214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f50009a70 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5ed9d640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 0x7f1f6009f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f5dd9b640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c730 tx=0x7f1f4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48005ab0 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f48005c10 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f58009400 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48001d30 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.399+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6009ee20 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.401+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6009f620 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.401+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1f48007070 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.401+0000 7f1f56ffd640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.401+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f1f480894c0 con 0x7f1f600d1000 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.402+0000 7f1f5e59c640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:59.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.402+0000 7f1f5e59c640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 0x7f1f4006dd00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f500114e0 tx=0x7f1f5000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.406+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1f6009f620 con 0x7f1f600d1000 2024-03-24T10:34:59.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.624+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f6015e770 con 0x7f1f600d1000 2024-03-24T10:34:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.625+0000 7f1f56ffd640 1 -- 172.21.15.1:0/231621214 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5163 (secure 0 0 0) 0x7f1f6015e770 con 0x7f1f600d1000 2024-03-24T10:34:59.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:34:59.625 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:34:56.717691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:34:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-03-24T10:34:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 msgr2=0x7f1f4006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 0x7f1f4006dd00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f500114e0 tx=0x7f1f5000b040 comp rx=0 tx=0).stop 2024-03-24T10:34:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 msgr2=0x7f1f60156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c730 tx=0x7f1f4800cbf0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 shutdown_connections 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f6015e390 0x7f1f6009f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f600d1000 0x7f1f60156cb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 --2- 172.21.15.1:0/231621214 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1f600b4ba0 0x7f1f601562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 >> 172.21.15.1:0/231621214 conn(0x7f1f600d3ec0 msgr2=0x7f1f600b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 shutdown_connections 2024-03-24T10:34:59.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:34:59.627+0000 7f1f64ad4640 1 -- 172.21.15.1:0/231621214 wait complete. 2024-03-24T10:35:00.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:34:53.709456+0000', 'modified': '2024-03-24T10:34:56.717691+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5280}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5280': {'gid': 5280, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/1351159218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 1351159218}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 1351159218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5280, 'qdb_cluster': [5280]} 2024-03-24T10:35:00.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T10:35:00.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T10:35:00.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.159+0000 7fec76233640 1 Processor -- start 2024-03-24T10:35:00.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.159+0000 7fec76233640 1 -- start start 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec76233640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec700d2bb0 0x7fec700b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec76233640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7015e770 con 0x7fec700d1000 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec76233640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec700d4e40 con 0x7fec7015e390 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec76233640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fec700b3190 con 0x7fec700d2bb0 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec700d2bb0 0x7fec700b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec700d2bb0 0x7fec700b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47680/0 (socket says 172.21.15.1:47680) 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec700d1000 0x7fec700b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60930/0 (socket says 172.21.15.1:60930) 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 -- 172.21.15.1:0/309117352 learned_addr learned my addr 172.21.15.1:0/309117352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 -- 172.21.15.1:0/309117352 >> v1:172.21.15.3:6789/0 conn(0x7fec7015e390 legacy=0x7fec700d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 -- 172.21.15.1:0/309117352 >> v1:172.21.15.1:6789/0 conn(0x7fec700d1000 legacy=0x7fec700b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6e7fc640 1 -- 172.21.15.1:0/309117352 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1927350821 0 0) 0x7fec700d4e40 con 0x7fec7015e390 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.160+0000 7fec6effd640 1 -- 172.21.15.1:0/309117352 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec700d1e40 con 0x7fec700d2bb0 2024-03-24T10:35:00.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.161+0000 7fec6effd640 1 --2- 172.21.15.1:0/309117352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d2bb0 0x7fec700b3b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fec64009870 tx=0x7fec6402b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5893c54338e7bec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.161+0000 7fec6e7fc640 1 -- 172.21.15.1:0/309117352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64030710 con 0x7fec700d2bb0 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.161+0000 7fec6e7fc640 1 -- 172.21.15.1:0/309117352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6402e050 con 0x7fec700d2bb0 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.161+0000 7fec6e7fc640 1 -- 172.21.15.1:0/309117352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64030a90 con 0x7fec700d2bb0 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 -- 172.21.15.1:0/309117352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d2bb0 msgr2=0x7fec700b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 --2- 172.21.15.1:0/309117352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d2bb0 0x7fec700b3b90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fec64009870 tx=0x7fec6402b740 comp rx=0 tx=0).stop 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 -- 172.21.15.1:0/309117352 shutdown_connections 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 --2- 172.21.15.1:0/309117352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d2bb0 0x7fec700b3b90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 -- 172.21.15.1:0/309117352 >> 172.21.15.1:0/309117352 conn(0x7fec700d3ec0 msgr2=0x7fec700d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 -- 172.21.15.1:0/309117352 shutdown_connections 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.162+0000 7fec76233640 1 -- 172.21.15.1:0/309117352 wait complete. 2024-03-24T10:35:00.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.163+0000 7fec76233640 1 Processor -- start 2024-03-24T10:35:00.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.163+0000 7fec76233640 1 -- start start 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec700d2bb0 0x7fec700cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec7015e390 0x7fec7006cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec7014f8c0 con 0x7fec700d1000 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fec700b6c10 con 0x7fec700d2bb0 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec76233640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fec700b6ef0 con 0x7fec7015e390 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47686/0 (socket says 172.21.15.1:47686) 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 -- 172.21.15.1:0/2286636822 learned_addr learned my addr 172.21.15.1:0/2286636822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6ffff640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec7015e390 0x7fec7006cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6effd640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec700d2bb0 0x7fec700cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 -- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec7015e390 msgr2=0x7fec7006cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec7015e390 0x7fec7006cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 -- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec700d2bb0 msgr2=0x7fec700cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec700d2bb0 0x7fec700cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 -- 172.21.15.1:0/2286636822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec640094d0 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c410 tx=0x7fec6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.164+0000 7fec57fff640 1 -- 172.21.15.1:0/2286636822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec600035f0 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.165+0000 7fec57fff640 1 -- 172.21.15.1:0/2286636822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec60003770 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.165+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec700b71d0 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.165+0000 7fec57fff640 1 -- 172.21.15.1:0/2286636822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60003b20 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.165+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec700b7790 con 0x7fec700d1000 2024-03-24T10:35:00.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.166+0000 7fec57fff640 1 -- 172.21.15.1:0/2286636822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fec60003c80 con 0x7fec700d1000 2024-03-24T10:35:00.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec57fff640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 0x7fec5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec6effd640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 0x7fec5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec57fff640 1 -- 172.21.15.1:0/2286636822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fec600897b0 con 0x7fec700d1000 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec6effd640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 0x7fec5c06dcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec64034000 tx=0x7fec64008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec78e0b740 1 Processor -- start 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.167+0000 7fec78e0b740 1 -- start start 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489899f0 0x55824898c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 0x55824898c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5582488d0180 con 0x558248989610 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5582488cd6b0 con 0x55824895caf0 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5582488c8e90 con 0x5582489899f0 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47702/0 (socket says 172.21.15.1:47702) 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.168+0000 7fec6f7fe640 1 -- 172.21.15.1:0/402348904 learned_addr learned my addr 172.21.15.1:0/402348904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6effd640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489899f0 0x55824898c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6ffff640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 0x55824898c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 -- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489899f0 msgr2=0x55824898c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489899f0 0x55824898c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 -- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 msgr2=0x55824898c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 0x55824898c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 -- 172.21.15.1:0/402348904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558248885d80 con 0x558248989610 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6ffff640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 0x55824898c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fec600601a0 tx=0x7fec600586c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3a5a1f04fc4891 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec557fa640 1 -- 172.21.15.1:0/402348904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec600606d0 con 0x558248989610 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec557fa640 1 -- 172.21.15.1:0/402348904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec600562c0 con 0x558248989610 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.169+0000 7fec557fa640 1 -- 172.21.15.1:0/402348904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60057410 con 0x558248989610 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- 172.21.15.1:0/402348904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 msgr2=0x55824895c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fec600601a0 tx=0x7fec600586c0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- 172.21.15.1:0/402348904 shutdown_connections 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55824895caf0 0x55824898c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489899f0 0x55824898c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 --2- 172.21.15.1:0/402348904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x55824895c5b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- 172.21.15.1:0/402348904 >> 172.21.15.1:0/402348904 conn(0x55824895e820 msgr2=0x5582489a73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- 172.21.15.1:0/402348904 shutdown_connections 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- 172.21.15.1:0/402348904 wait complete. 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.170+0000 7fec78e0b740 1 -- start start 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 Processor -- start 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558248989610 0x558248970500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248984f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55824886dd50 con 0x5582489899f0 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x55824859c310 con 0x558248989610 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x55824884e8f0 con 0x5582489a7a40 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec78e0b740 10 client.0 _ll_register_callbacks cb 0x5582485d9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47714/0 (socket says 172.21.15.1:47714) 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.171+0000 7fec6effd640 1 -- 172.21.15.1:0/3095147991 learned_addr learned my addr 172.21.15.1:0/3095147991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558248989610 0x558248970500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6ffff640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248984f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 -- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 msgr2=0x558248984f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248984f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 -- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558248989610 msgr2=0x558248970500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558248989610 0x558248970500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 -- 172.21.15.1:0/3095147991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec700b7790 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec6effd640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec6402bd50 tx=0x7fec6402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287af59f1f3c09c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64050410 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fec64050570 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5582485c86c0 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fec64050730 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec78e0b740 10 client.5463 fetch_fsmap learned FSMap version 16 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.172+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x5582488c7510 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5582485c86c0 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7fec64044040 con 0x5582489899f0 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 10 client.5463 fetch_fsmap finished waiting for FSMap version 16 2024-03-24T10:35:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 10 client.5463 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 20 client.5463 populate_metadata read hostname 'smithi001' 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x55824898aa90 0x558248972560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 4 client.5463 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 --> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558248619270 con 0x55824898aa90 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x55824898aa90 0x558248972560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec4e7fc640 10 client.5463 ms_handle_connect on v2:172.21.15.1:6836/1351159218 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.173+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x55824898aa90 0x558248972560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d567bde0c62a716e server_cookie=1197ce0dc453d2c in_seq=0 out_seq=0 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec4e7fc640 1 -- 172.21.15.1:0/3095147991 <== mds.0 v2:172.21.15.1:6836/1351159218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fec60076e90 con 0x55824898aa90 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec4e7fc640 10 client.5463 handle_command_reply: tid=0 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 client.5463 shutdown 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 msgr2=0x5582489849e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fec6402bd50 tx=0x7fec6402d000 comp rx=0 tx=0).stop 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 shutdown_connections 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x55824898aa90 0x558248972560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248984f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x5582489849e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec4e7fc640 0 client.5463 ms_handle_reset on v2:172.21.15.1:6836/1351159218 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3095147991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558248989610 0x558248970500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 >> 172.21.15.1:0/3095147991 conn(0x55824895caf0 msgr2=0x5582489633e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 shutdown_connections 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 1 -- 172.21.15.1:0/3095147991 wait complete. 2024-03-24T10:35:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.174+0000 7fec78e0b740 20 client.5463 trim_cache size 0 max 16384 2024-03-24T10:35:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.210+0000 7fec78e0b740 1 Processor -- start 2024-03-24T10:35:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.210+0000 7fec78e0b740 1 -- start start 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489899f0 0x558248960110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248960650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5582489b6460 con 0x558248989610 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5582488d0650 con 0x5582489899f0 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x5582489824d0 con 0x5582489a7a40 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47728/0 (socket says 172.21.15.1:47728) 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 -- 172.21.15.1:0/3584593161 learned_addr learned my addr 172.21.15.1:0/3584593161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6effd640 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489899f0 0x558248960110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 -- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 msgr2=0x558248960650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248960650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 -- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489899f0 msgr2=0x558248960110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489899f0 0x558248960110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.211+0000 7fec6f7fe640 1 -- 172.21.15.1:0/3584593161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558248863a80 con 0x558248989610 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fec60054420 tx=0x7fec60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae056d3114a82fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec54ff9640 1 -- 172.21.15.1:0/3584593161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60078ea0 con 0x558248989610 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec54ff9640 1 -- 172.21.15.1:0/3584593161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec60076e90 con 0x558248989610 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec54ff9640 1 -- 172.21.15.1:0/3584593161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60075040 con 0x558248989610 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec78e0b740 1 -- 172.21.15.1:0/3584593161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 msgr2=0x558248970150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.212+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fec60054420 tx=0x7fec60017000 comp rx=0 tx=0).stop 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 -- 172.21.15.1:0/3584593161 shutdown_connections 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x5582489a7a40 0x558248960650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489899f0 0x558248960110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 --2- 172.21.15.1:0/3584593161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558248989610 0x558248970150 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 -- 172.21.15.1:0/3584593161 >> 172.21.15.1:0/3584593161 conn(0x55824895caf0 msgr2=0x55824898c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 -- 172.21.15.1:0/3584593161 shutdown_connections 2024-03-24T10:35:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 -- 172.21.15.1:0/3584593161 wait complete. 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.213+0000 7fec78e0b740 1 -- start start 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.214+0000 7fec78e0b740 1 Processor -- start 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 0x558248a209c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489a7a40 0x558248960550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558248a20f00 con 0x5582489899f0 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x5582489ca630 con 0x5582489a7a40 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x558248961430 con 0x558248989610 2024-03-24T10:35:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec78e0b740 10 client.0 _ll_register_callbacks cb 0x7fec64035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47738/0 (socket says 172.21.15.1:47738) 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 -- 172.21.15.1:0/570870870 learned_addr learned my addr 172.21.15.1:0/570870870 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6ffff640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489a7a40 0x558248960550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 0x558248a209c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 -- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 msgr2=0x558248a209c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 0x558248a209c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 -- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489a7a40 msgr2=0x558248960550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489a7a40 0x558248960550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 -- 172.21.15.1:0/570870870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec60085040 con 0x5582489899f0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 0x558248a209c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec6effd640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fec64009770 tx=0x7fec64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd735ac54ae3245 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.215+0000 7fec4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64004450 con 0x5582489899f0 2024-03-24T10:35:00.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fec64003610 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec600562c0 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fec6403cb50 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 10 client.5469 fetch_fsmap learned FSMap version 16 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x5582488897e0 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec600562c0 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7fec64044070 con 0x5582489899f0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 10 client.5469 fetch_fsmap finished waiting for FSMap version 16 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 10 client.5469 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 20 client.5469 populate_metadata read hostname 'smithi001' 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x5582489c9ae0 0x55824898ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 4 client.5469 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 --> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] -- command(tid 0: {"prefix": "status"}) -- 0x558248605610 con 0x5582489c9ae0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x5582489c9ae0 0x55824898ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.216+0000 7fec4e7fc640 10 client.5469 ms_handle_connect on v2:172.21.15.1:6836/1351159218 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec6f7fe640 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x5582489c9ae0 0x55824898ae70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c4d12845ec2ef8c server_cookie=342db84cfa876614 in_seq=0 out_seq=0 2024-03-24T10:35:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec4e7fc640 1 -- 172.21.15.1:0/570870870 <== mds.0 v2:172.21.15.1:6836/1351159218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558248605610 con 0x5582489c9ae0 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec4e7fc640 10 client.5469 handle_command_reply: tid=0 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 client.5469 shutdown 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 msgr2=0x558248960010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fec64009770 tx=0x7fec64004730 comp rx=0 tx=0).stop 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 shutdown_connections 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:6836/1351159218,v1:172.21.15.1:6837/1351159218] conn(0x5582489c9ae0 0x55824898ae70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582489a7a40 0x558248960550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5582489899f0 0x558248960010 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec4e7fc640 0 client.5469 ms_handle_reset on v2:172.21.15.1:6836/1351159218 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 --2- 172.21.15.1:0/570870870 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x558248989610 0x558248a209c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 >> 172.21.15.1:0/570870870 conn(0x55824895caf0 msgr2=0x5582489aaf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 shutdown_connections 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 1 -- 172.21.15.1:0/570870870 wait complete. 2024-03-24T10:35:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec78e0b740 20 client.5469 trim_cache size 0 max 16384 2024-03-24T10:35:00.223 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:35:00.223 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "a0192ab9-51d5-4157-8f81-4a83111a7dfa", 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5280, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.8758280330000003, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 34, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 34, 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13.408478155999999 2024-03-24T10:35:00.224 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:35:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 msgr2=0x7fec5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 0x7fec5c06dcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec64034000 tx=0x7fec64008030 comp rx=0 tx=0).stop 2024-03-24T10:35:00.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 msgr2=0x7fec7014f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.217+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c410 tx=0x7fec6000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 shutdown_connections 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fec5c06b810 0x7fec5c06dcd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec7015e390 0x7fec7006cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec700d2bb0 0x7fec700cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 --2- 172.21.15.1:0/2286636822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec700d1000 0x7fec7014f380 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 >> 172.21.15.1:0/2286636822 conn(0x7fec700d3ec0 msgr2=0x7fec700b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 shutdown_connections 2024-03-24T10:35:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.218+0000 7fec76233640 1 -- 172.21.15.1:0/2286636822 wait complete. 2024-03-24T10:35:00.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:35:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.330+0000 7f697d95e640 1 Processor -- start 2024-03-24T10:35:00.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.330+0000 7f697d95e640 1 -- start start 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f697d95e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f697806d2d0 0x7f697815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f697d95e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69780d10e0 con 0x7f697806c770 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f697d95e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69780b4c70 con 0x7f69780b74c0 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f697d95e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f69780d1340 con 0x7f697806d2d0 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f697806d2d0 0x7f697815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f697806d2d0 0x7f697815d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47748/0 (socket says 172.21.15.1:47748) 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 -- 172.21.15.1:0/4268077860 learned_addr learned my addr 172.21.15.1:0/4268077860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f696e7fc640 1 --1- 172.21.15.1:0/4268077860 >> v1:172.21.15.1:6789/0 conn(0x7f697806c770 0x7f697806cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60934/0 (socket says 172.21.15.1:60934) 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 -- 172.21.15.1:0/4268077860 >> v1:172.21.15.3:6789/0 conn(0x7f69780b74c0 legacy=0x7f69780d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 -- 172.21.15.1:0/4268077860 >> v1:172.21.15.1:6789/0 conn(0x7f697806c770 legacy=0x7f697806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69777fe640 1 -- 172.21.15.1:0/4268077860 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69780b4f50 con 0x7f697806d2d0 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.331+0000 7f69767fc640 1 -- 172.21.15.1:0/4268077860 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1351654865 0 0) 0x7f69780d10e0 con 0x7f697806c770 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.332+0000 7f69777fe640 1 --2- 172.21.15.1:0/4268077860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806d2d0 0x7f697815d8f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6970009e50 tx=0x7f697002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2ddbac5cae9a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.332+0000 7f69767fc640 1 -- 172.21.15.1:0/4268077860 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697002cbb0 con 0x7f697806d2d0 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.332+0000 7f69767fc640 1 -- 172.21.15.1:0/4268077860 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697002cd10 con 0x7f697806d2d0 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.332+0000 7f69767fc640 1 -- 172.21.15.1:0/4268077860 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970034610 con 0x7f697806d2d0 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 -- 172.21.15.1:0/4268077860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806d2d0 msgr2=0x7f697815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 --2- 172.21.15.1:0/4268077860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806d2d0 0x7f697815d8f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6970009e50 tx=0x7f697002c010 comp rx=0 tx=0).stop 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 -- 172.21.15.1:0/4268077860 shutdown_connections 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 --2- 172.21.15.1:0/4268077860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806d2d0 0x7f697815d8f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 -- 172.21.15.1:0/4268077860 >> 172.21.15.1:0/4268077860 conn(0x7f69780d1c80 msgr2=0x7f69780d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 -- 172.21.15.1:0/4268077860 shutdown_connections 2024-03-24T10:35:00.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.333+0000 7f697d95e640 1 -- 172.21.15.1:0/4268077860 wait complete. 2024-03-24T10:35:00.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.334+0000 7f697d95e640 1 Processor -- start 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.334+0000 7f697d95e640 1 -- start start 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f697806d2d0 0x7f697815d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 0x7f697809e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f697815e600 con 0x7f697806c770 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f697814c580 con 0x7f69780b74c0 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f697d95e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f697815e040 con 0x7f697806d2d0 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47762/0 (socket says 172.21.15.1:47762) 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 -- 172.21.15.1:0/2164593408 learned_addr learned my addr 172.21.15.1:0/2164593408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:00.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f69777fe640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 0x7f697809e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f696e7fc640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f697806d2d0 0x7f697815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 -- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f697806d2d0 msgr2=0x7f697815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f697806d2d0 0x7f697815d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 -- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 msgr2=0x7f697809e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 0x7f697809e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 -- 172.21.15.1:0/2164593408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6970009b30 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f69777fe640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 0x7f697809e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6976ffd640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f696800c450 tx=0x7f696800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69680052d0 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6968005430 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.335+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968012870 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.336+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697815e320 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.336+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697814bd10 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.337+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69780b6e70 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.337+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f69680129d0 con 0x7f697806c770 2024-03-24T10:35:00.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.338+0000 7f6974ff9640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 0x7f695806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:00.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.338+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f6968089d10 con 0x7f697806c770 2024-03-24T10:35:00.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.338+0000 7f696e7fc640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 0x7f695806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:00.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.338+0000 7f696e7fc640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 0x7f695806dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f69781f7290 tx=0x7f6960007d00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:00.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.346+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6968054730 con 0x7f697806c770 2024-03-24T10:35:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.565+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f697804f4e0 con 0x7f697806c770 2024-03-24T10:35:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.566+0000 7f6974ff9640 1 -- 172.21.15.1:0/2164593408 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f6968057df0 con 0x7f697806c770 2024-03-24T10:35:00.566 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 msgr2=0x7f695806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 0x7f695806dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f69781f7290 tx=0x7f6960007d00 comp rx=0 tx=0).stop 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 msgr2=0x7f697814b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f696800c450 tx=0x7f696800c910 comp rx=0 tx=0).stop 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 shutdown_connections 2024-03-24T10:35:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b810 0x7f695806dcd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69780b74c0 0x7f697809e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f697806d2d0 0x7f697815d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 --2- 172.21.15.1:0/2164593408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f697806c770 0x7f697814b7d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 >> 172.21.15.1:0/2164593408 conn(0x7f69780d1c80 msgr2=0x7f69780c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 shutdown_connections 2024-03-24T10:35:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00.568+0000 7f697d95e640 1 -- 172.21.15.1:0/2164593408 wait complete. 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:35:00.579 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:00.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:00.580 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:00.596 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42640sec preferred_lft 42640sec 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:35:00.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:00.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:00.599 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:35:00.661 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:35:00.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:00.662 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:35:00.718 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:35:00.718 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:35:00.718 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:35:00.718 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:35:00.718 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:35:00.719 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:35:00.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:35:00.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:00.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:00.853 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:35:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:35:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:35:00.869 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:00.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:00.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:35:00.935 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:35:00.935 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:00.936 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:00.989 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:00.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:00.990 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:01.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:01.111 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:01.166 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:01.167 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:01.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:01.168 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:01.220 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:35:01.220 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:35:01.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:01.290 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:01.346 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:01.347 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:01.348 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:01.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:01.349 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:01.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:35:01.349+0000 7f4b3da0c480 -1 init, newargv = 0x55cd19b9da70 newargc=15 2024-03-24T10:35:01.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[52115]: starting ceph client 2024-03-24T10:35:01.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[52115]: starting fuse 2024-03-24T10:35:01.376 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:35:01.377 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:01.377 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:35:01.378 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:35:01.504 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.52115.asok 2024-03-24T10:35:01.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:01.515 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52115.asok 2024-03-24T10:35:01.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:01.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52115.asok status 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:35:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "52115", 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5484, 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5484 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1220064988 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1220064988 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.5484 192.168.0.1:0/1220064988", 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1220064988", 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 16, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 34, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:35:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:35:01.657 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:35:01.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:01.665 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:35:01.665 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:01.680 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:35:01.680 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:01.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:01.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:01.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-24T10:35:01.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.894+0000 7f0904c07640 1 Processor -- start 2024-03-24T10:35:01.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.894+0000 7f0904c07640 1 -- start start 2024-03-24T10:35:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f0904c07640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f090015e860 0x7f09000cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f0904c07640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09000d04c0 0x7f090006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f0904c07640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09000b45c0 con 0x7f09000cbf90 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f0904c07640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f090006c930 con 0x7f09000d04c0 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f0904c07640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f090015ed00 con 0x7f090015e860 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09000d04c0 0x7f090006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09000d04c0 0x7f090006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35084/0 (socket says 172.21.15.1:35084) 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 -- 172.21.15.1:0/1519731914 learned_addr learned my addr 172.21.15.1:0/1519731914 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08f7fff640 1 --2- 172.21.15.1:0/1519731914 >> v2:172.21.15.3:3300/0 conn(0x7f090015e860 0x7f09000cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 -- 172.21.15.1:0/1519731914 >> v2:172.21.15.3:3300/0 conn(0x7f090015e860 msgr2=0x7f09000cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 --2- 172.21.15.1:0/1519731914 >> v2:172.21.15.3:3300/0 conn(0x7f090015e860 0x7f09000cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 -- 172.21.15.1:0/1519731914 >> v1:172.21.15.3:6789/0 conn(0x7f09000cbf90 legacy=0x7f090015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.895+0000 7f08fed9d640 1 -- 172.21.15.1:0/1519731914 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09000d0920 con 0x7f09000d04c0 2024-03-24T10:35:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f08fed9d640 1 --2- 172.21.15.1:0/1519731914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f090006cd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0900072d20 tx=0x7f08f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8616c06267357b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f08fdd9b640 1 -- 172.21.15.1:0/1519731914 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f802ca80 con 0x7f09000d04c0 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f08fdd9b640 1 -- 172.21.15.1:0/1519731914 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f802cbe0 con 0x7f09000d04c0 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f08fdd9b640 1 -- 172.21.15.1:0/1519731914 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f80343f0 con 0x7f09000d04c0 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f0904c07640 1 -- 172.21.15.1:0/1519731914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 msgr2=0x7f090006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f0904c07640 1 --2- 172.21.15.1:0/1519731914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f090006cd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0900072d20 tx=0x7f08f802bee0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f0904c07640 1 -- 172.21.15.1:0/1519731914 shutdown_connections 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f0904c07640 1 --2- 172.21.15.1:0/1519731914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f090006cd50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.896+0000 7f0904c07640 1 --2- 172.21.15.1:0/1519731914 >> v2:172.21.15.3:3300/0 conn(0x7f090015e860 0x7f09000cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.897+0000 7f0904c07640 1 -- 172.21.15.1:0/1519731914 >> 172.21.15.1:0/1519731914 conn(0x7f090006b4f0 msgr2=0x7f090006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.897+0000 7f0904c07640 1 -- 172.21.15.1:0/1519731914 shutdown_connections 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.897+0000 7f0904c07640 1 -- 172.21.15.1:0/1519731914 wait complete. 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.898+0000 7f0904c07640 1 Processor -- start 2024-03-24T10:35:01.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.898+0000 7f0904c07640 1 -- start start 2024-03-24T10:35:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 0x7f090014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f090015e860 0x7f0900157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0900155a90 con 0x7f09000d04c0 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0900155d70 con 0x7f090015e860 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f0904c07640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0900156050 con 0x7f09000cbf90 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35092/0 (socket says 172.21.15.1:35092) 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 -- 172.21.15.1:0/2135484791 learned_addr learned my addr 172.21.15.1:0/2135484791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08fed9d640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f090015e860 0x7f0900157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08fe59c640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 0x7f090014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 -- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 msgr2=0x7f090014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 0x7f090014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 -- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f090015e860 msgr2=0x7f0900157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f090015e860 0x7f0900157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08f7fff640 1 -- 172.21.15.1:0/2135484791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f8009b30 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.899+0000 7f08fe59c640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 0x7f090014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f08f7fff640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f08e800d800 tx=0x7f08e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e800b6c0 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e800b820 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0900156330 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e800bbb0 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.900+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090014bb50 con 0x7f09000d04c0 2024-03-24T10:35:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.901+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f090004f470 con 0x7f09000d04c0 2024-03-24T10:35:01.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.901+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f08e800bd10 con 0x7f09000d04c0 2024-03-24T10:35:01.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.902+0000 7f08f6ffd640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 0x7f08e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:01.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.902+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f08e808a6b0 con 0x7f09000d04c0 2024-03-24T10:35:01.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.902+0000 7f08fe59c640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 0x7f08e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:01.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.902+0000 7f08fe59c640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 0x7f08e006dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f08f0001660 tx=0x7f08f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:01.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:01.906+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08e80550d0 con 0x7f09000d04c0 2024-03-24T10:35:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.208+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7f09000b3b40 con 0x7f09000d04c0 2024-03-24T10:35:02.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.382+0000 7f08f6ffd640 1 -- 172.21.15.1:0/2135484791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v35) ==== 124+0+0 (secure 0 0 0) 0x7f08e8058790 con 0x7f09000d04c0 2024-03-24T10:35:02.382 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'data_new' created 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.384+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 msgr2=0x7f08e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 0x7f08e006dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f08f0001660 tx=0x7f08f0009040 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 msgr2=0x7f0900156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f08e800d800 tx=0x7f08e800dcc0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 shutdown_connections 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f08e006b810 0x7f08e006dcd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f090015e860 0x7f0900157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09000d04c0 0x7f0900156b30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 --2- 172.21.15.1:0/2135484791 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f09000cbf90 0x7f090014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 >> 172.21.15.1:0/2135484791 conn(0x7f090006b4f0 msgr2=0x7f0900072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:02.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 shutdown_connections 2024-03-24T10:35:02.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.385+0000 7f0904c07640 1 -- 172.21.15.1:0/2135484791 wait complete. 2024-03-24T10:35:02.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-24T10:35:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.504+0000 7fb164095640 1 Processor -- start 2024-03-24T10:35:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.504+0000 7fb164095640 1 -- start start 2024-03-24T10:35:02.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15c1615f0 0x7fb15c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb15c0b4e40 0x7fb15c161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb15c0cfd50 con 0x7fb15c1615f0 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb15c0c7a40 con 0x7fb15c0b4e40 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb164095640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb15c0b55b0 con 0x7fb15c1619d0 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb161e31640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15c1615f0 0x7fb15c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb161e31640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15c1615f0 0x7fb15c0d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35106/0 (socket says 172.21.15.1:35106) 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb161e31640 1 -- 172.21.15.1:0/1946641763 learned_addr learned my addr 172.21.15.1:0/1946641763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb161630640 1 --2- 172.21.15.1:0/1946641763 >> v2:172.21.15.3:3300/0 conn(0x7fb15c0b4e40 0x7fb15c161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.505+0000 7fb162632640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.506+0000 7fb161e31640 1 -- 172.21.15.1:0/1946641763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 msgr2=0x7fb15c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:02.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.506+0000 7fb161e31640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.506+0000 7fb161e31640 1 -- 172.21.15.1:0/1946641763 >> v2:172.21.15.3:3300/0 conn(0x7fb15c0b4e40 msgr2=0x7fb15c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:02.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.506+0000 7fb161e31640 1 --2- 172.21.15.1:0/1946641763 >> v2:172.21.15.3:3300/0 conn(0x7fb15c0b4e40 0x7fb15c161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.506+0000 7fb161e31640 1 -- 172.21.15.1:0/1946641763 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15c0b52d0 con 0x7fb15c1615f0 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.507+0000 7fb161e31640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1615f0 0x7fb15c0d4a10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0737b0 tx=0x7fb15802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a465db57a78a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.507+0000 7fb160e2f640 1 -- 172.21.15.1:0/1946641763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031640 con 0x7fb15c1615f0 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.507+0000 7fb160e2f640 1 -- 172.21.15.1:0/1946641763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1580317a0 con 0x7fb15c1615f0 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.507+0000 7fb160e2f640 1 -- 172.21.15.1:0/1946641763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031b30 con 0x7fb15c1615f0 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 -- 172.21.15.1:0/1946641763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1615f0 msgr2=0x7fb15c0d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1615f0 0x7fb15c0d4a10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0737b0 tx=0x7fb15802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 -- 172.21.15.1:0/1946641763 shutdown_connections 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 --2- 172.21.15.1:0/1946641763 >> v2:172.21.15.3:3300/0 conn(0x7fb15c0b4e40 0x7fb15c161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 --2- 172.21.15.1:0/1946641763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1615f0 0x7fb15c0d4a10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 -- 172.21.15.1:0/1946641763 >> 172.21.15.1:0/1946641763 conn(0x7fb15c0d3ef0 msgr2=0x7fb15c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 -- 172.21.15.1:0/1946641763 shutdown_connections 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.508+0000 7fb164095640 1 -- 172.21.15.1:0/1946641763 wait complete. 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.509+0000 7fb164095640 1 Processor -- start 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.509+0000 7fb164095640 1 -- start start 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb15c1615f0 0x7fb15c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c1f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c1f4bb0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c157cb0 con 0x7fb15c1615f0 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb164095640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb15c1f6f00 con 0x7fb15c1619d0 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35118/0 (socket says 172.21.15.1:35118) 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 -- 172.21.15.1:0/565017016 learned_addr learned my addr 172.21.15.1:0/565017016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb161e31640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb15c1615f0 0x7fb15c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb161630640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c1f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 -- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 msgr2=0x7fb15c1f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c1f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 -- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb15c1615f0 msgr2=0x7fb15c157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb15c1615f0 0x7fb15c157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 -- 172.21.15.1:0/565017016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1580094d0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.510+0000 7fb162632640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb15400cc30 tx=0x7fb15400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.511+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15400ae80 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.511+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb15400ce40 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.511+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15c1576f0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.511+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154003b90 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.511+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb15c0b8480 con 0x7fb15c1f4d70 2024-03-24T10:35:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.512+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb15c0b82e0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.512+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb15400f3f0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.513+0000 7fb1527fc640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 0x7fb14006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:02.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.513+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6072+0+0 (secure 0 0 0) 0x7fb15408a560 con 0x7fb15c1f4d70 2024-03-24T10:35:02.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.513+0000 7fb161e31640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 0x7fb14006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:02.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.513+0000 7fb161e31640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 0x7fb14006dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb158036000 tx=0x7fb15803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:02.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.517+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb15c0b82e0 con 0x7fb15c1f4d70 2024-03-24T10:35:02.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:02.916+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7fb15c0b82e0 con 0x7fb15c1f4d70 2024-03-24T10:35:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.390+0000 7fb1527fc640 1 -- 172.21.15.1:0/565017016 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 6 to fsmap v17) ==== 131+0+0 (secure 0 0 0) 0x7fb15c0b82e0 con 0x7fb15c1f4d70 2024-03-24T10:35:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:added data pool 6 to fsmap 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.395+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 msgr2=0x7fb14006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.395+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 0x7fb14006dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb158036000 tx=0x7fb15803c040 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.395+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 msgr2=0x7fb15c1f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.395+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb15400cc30 tx=0x7fb15400a5c0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 shutdown_connections 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb14006b810 0x7fb14006dcd0 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb158036000 tx=0x7fb15803c040 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15c1f4d70 0x7fb15c1f69c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb15c1619d0 0x7fb15c1f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 --2- 172.21.15.1:0/565017016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb15c1615f0 0x7fb15c157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.396+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 >> 172.21.15.1:0/565017016 conn(0x7fb15c0d3ef0 msgr2=0x7fb15c0ce9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.397+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 shutdown_connections 2024-03-24T10:35:03.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.397+0000 7fb164095640 1 -- 172.21.15.1:0/565017016 wait complete. 2024-03-24T10:35:03.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.512+0000 7fe49fdd0640 1 Processor -- start 2024-03-24T10:35:03.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.512+0000 7fe49fdd0640 1 -- start start 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 0x7fe4980d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980d1000 0x7fe4980d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4980b3320 0x7fe4980cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe4980b3190 con 0x7fe4980b3320 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe49815e770 con 0x7fe49815e390 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49fdd0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe4980d4e40 con 0x7fe4980d1000 2024-03-24T10:35:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4980b3320 0x7fe4980cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4980b3320 0x7fe4980cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35134/0 (socket says 172.21.15.1:35134) 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 -- 172.21.15.1:0/1146505778 learned_addr learned my addr 172.21.15.1:0/1146505778 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49db6c640 1 --2- 172.21.15.1:0/1146505778 >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 0x7fe4980d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 -- 172.21.15.1:0/1146505778 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980d1000 msgr2=0x7fe4980d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49d36b640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980d1000 0x7fe4980d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980d1000 0x7fe4980d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 -- 172.21.15.1:0/1146505778 >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 msgr2=0x7fe4980d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 --2- 172.21.15.1:0/1146505778 >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 0x7fe4980d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49e36d640 1 -- 172.21.15.1:0/1146505778 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4980d1e40 con 0x7fe4980b3320 2024-03-24T10:35:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.513+0000 7fe49db6c640 1 --2- 172.21.15.1:0/1146505778 >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 0x7fe4980d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.514+0000 7fe49e36d640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4980b3320 0x7fe4980cea60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe4980d3040 tx=0x7fe49002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b7f57455c8eda6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49cb6a640 1 -- 172.21.15.1:0/1146505778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49002c950 con 0x7fe4980b3320 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49cb6a640 1 -- 172.21.15.1:0/1146505778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49002cab0 con 0x7fe4980b3320 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49cb6a640 1 -- 172.21.15.1:0/1146505778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49002ce40 con 0x7fe4980b3320 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 -- 172.21.15.1:0/1146505778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4980b3320 msgr2=0x7fe4980cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4980b3320 0x7fe4980cea60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe4980d3040 tx=0x7fe49002bdb0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 -- 172.21.15.1:0/1146505778 shutdown_connections 2024-03-24T10:35:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4980b3320 0x7fe4980cea60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/1146505778 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980d1000 0x7fe4980d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/1146505778 >> v2:172.21.15.3:3300/0 conn(0x7fe49815e390 0x7fe4980d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 -- 172.21.15.1:0/1146505778 >> 172.21.15.1:0/1146505778 conn(0x7fe4980d3ec0 msgr2=0x7fe4980d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 -- 172.21.15.1:0/1146505778 shutdown_connections 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.515+0000 7fe49fdd0640 1 -- 172.21.15.1:0/1146505778 wait complete. 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.516+0000 7fe49fdd0640 1 Processor -- start 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.516+0000 7fe49fdd0640 1 -- start start 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980b3320 0x7fe4981f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 0x7fe498155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe498156240 con 0x7fe49815e390 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe4980cc550 con 0x7fe4980d1000 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49fdd0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe4980cc830 con 0x7fe4980b3320 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35138/0 (socket says 172.21.15.1:35138) 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 -- 172.21.15.1:0/628738377 learned_addr learned my addr 172.21.15.1:0/628738377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49db6c640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980b3320 0x7fe4981f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49d36b640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 0x7fe498155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 -- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980b3320 msgr2=0x7fe4981f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980b3320 0x7fe4981f5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 -- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 msgr2=0x7fe498155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.517+0000 7fe49e36d640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 0x7fe498155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe49e36d640 1 -- 172.21.15.1:0/628738377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe490009b30 con 0x7fe49815e390 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe49d36b640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 0x7fe498155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe49e36d640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe49002bdf0 tx=0x7fe490004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49003e450 con 0x7fe49815e390 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49003e5b0 con 0x7fe49815e390 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4980ccb10 con 0x7fe49815e390 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49003e960 con 0x7fe49815e390 2024-03-24T10:35:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.518+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4980a1b40 con 0x7fe49815e390 2024-03-24T10:35:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.519+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4980cd030 con 0x7fe49815e390 2024-03-24T10:35:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.520+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe49003eac0 con 0x7fe49815e390 2024-03-24T10:35:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.520+0000 7fe48e7fc640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 0x7fe47c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.520+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fe4900ad320 con 0x7fe49815e390 2024-03-24T10:35:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.520+0000 7fe49db6c640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 0x7fe47c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.520+0000 7fe49db6c640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 0x7fe47c06dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe48800b7c0 tx=0x7fe488008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.524+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe4980cd030 con 0x7fe49815e390 2024-03-24T10:35:03.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.743+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4980cd030 con 0x7fe49815e390 2024-03-24T10:35:03.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.744+0000 7fe48e7fc640 1 -- 172.21.15.1:0/628738377 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5165 (secure 0 0 0) 0x7fe4980cd030 con 0x7fe49815e390 2024-03-24T10:35:03.744 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:03.744 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:35:02.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:35:03.744 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 msgr2=0x7fe47c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 0x7fe47c06dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe48800b7c0 tx=0x7fe488008040 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 msgr2=0x7fe4980a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe49002bdf0 tx=0x7fe490004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 shutdown_connections 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe47c06b810 0x7fe47c06dcd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe49815e390 0x7fe4980a1240 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4980d1000 0x7fe498155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 --2- 172.21.15.1:0/628738377 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe4980b3320 0x7fe4981f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 >> 172.21.15.1:0/628738377 conn(0x7fe4980d3ec0 msgr2=0x7fe49806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 shutdown_connections 2024-03-24T10:35:03.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.746+0000 7fe49fdd0640 1 -- 172.21.15.1:0/628738377 wait complete. 2024-03-24T10:35:03.760 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:03.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.861+0000 7f652122e640 1 Processor -- start 2024-03-24T10:35:03.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.861+0000 7f652122e640 1 -- start start 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f652122e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f651c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f652122e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f651c0b4ba0 0x7f651c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f652122e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651c15e770 con 0x7f651c0d1000 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f652122e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f651c0b3190 con 0x7f651c0b4ba0 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f652122e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f651c0d4e40 con 0x7f651c15e390 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f651a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f651c0d1000 0x7f651c0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45752/0 (socket says 172.21.15.1:45752) 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.862+0000 7f651a59c640 1 -- 172.21.15.1:0/3319733186 learned_addr learned my addr 172.21.15.1:0/3319733186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f651c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651b59e640 1 --2- 172.21.15.1:0/3319733186 >> v2:172.21.15.3:3300/0 conn(0x7f651c0b4ba0 0x7f651c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403477269 0 0) 0x7f6510001110 con 0x7f651c0d1000 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6504003520 con 0x7f651c0d1000 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1691197495 0 0) 0x7f6504003520 con 0x7f651c0d1000 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 msgr2=0x7f651c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f651c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 >> v2:172.21.15.3:3300/0 conn(0x7f651c0b4ba0 msgr2=0x7f651c0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 --2- 172.21.15.1:0/3319733186 >> v2:172.21.15.3:3300/0 conn(0x7f651c0b4ba0 0x7f651c0d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c0d1e40 con 0x7f651c0d1000 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f6510001800 con 0x7f651c0d1000 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 >> v1:172.21.15.1:6789/0 conn(0x7f651c0d1000 legacy=0x7f651c0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6504003970 0x7f6504003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65040047c0 0x7f6504004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f6504005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0d4e40 con 0x7f6504003970 2024-03-24T10:35:03.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f650400b980 con 0x7f65040047c0 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f650400bb40 con 0x7f651c15e390 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6504003970 0x7f6504003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651b59e640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f6504005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651a59c640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65040047c0 0x7f6504004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 -- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 msgr2=0x7f6504005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c15e390 0x7f6504005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 -- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65040047c0 msgr2=0x7f6504004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65040047c0 0x7f6504004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.863+0000 7f651ad9d640 1 -- 172.21.15.1:0/3319733186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6504003520 con 0x7f6504003970 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.864+0000 7f651ad9d640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6504003970 0x7f6504003d90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f650c009ac0 tx=0x7f65040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3553164fee701d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.864+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c02c6b0 con 0x7f6504003970 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.864+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f650c02c810 con 0x7f6504003970 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.864+0000 7f6519d9b640 1 -- 172.21.15.1:0/3319733186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c02cba0 con 0x7f6504003970 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 -- 172.21.15.1:0/3319733186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6504003970 msgr2=0x7f6504003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 --2- 172.21.15.1:0/3319733186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6504003970 0x7f6504003d90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f650c009ac0 tx=0x7f65040057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f651b59e640 1 -- 172.21.15.1:0/3319733186 reap_dead start 2024-03-24T10:35:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 -- 172.21.15.1:0/3319733186 shutdown_connections 2024-03-24T10:35:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 -- 172.21.15.1:0/3319733186 >> 172.21.15.1:0/3319733186 conn(0x7f651c0d3ec0 msgr2=0x7f651c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 -- 172.21.15.1:0/3319733186 shutdown_connections 2024-03-24T10:35:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.865+0000 7f652122e640 1 -- 172.21.15.1:0/3319733186 wait complete. 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.866+0000 7f652122e640 1 Processor -- start 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.866+0000 7f652122e640 1 -- start start 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c1523b0 0x7f651c1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 0x7f651c0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0b1450 con 0x7f651c0b0fd0 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1f4930 con 0x7f651c1523b0 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f652122e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f651c1f42c0 con 0x7f651c1f3c70 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f651b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f651b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35160/0 (socket says 172.21.15.1:35160) 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f651b59e640 1 -- 172.21.15.1:0/1177758026 learned_addr learned my addr 172.21.15.1:0/1177758026 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f651ad9d640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c1523b0 0x7f651c1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.867+0000 7f651a59c640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 0x7f651c0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 -- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 msgr2=0x7f651c0b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 0x7f651c0b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 -- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c1523b0 msgr2=0x7f651c1f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c1523b0 0x7f651c1f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 -- 172.21.15.1:0/1177758026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c1f45a0 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651a59c640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 0x7f651c0b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f651b59e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f651c0b5960 tx=0x7f651400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651400e040 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f651400f040 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c1f4b80 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651400fb70 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.868+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c152790 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.869+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c1f4050 con 0x7f651c0b0fd0 2024-03-24T10:35:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.869+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f651400fcd0 con 0x7f651c0b0fd0 2024-03-24T10:35:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.870+0000 7f65037fe640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 0x7f650806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.870+0000 7f651ad9d640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 0x7f650806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.870+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f651408a890 con 0x7f651c0b0fd0 2024-03-24T10:35:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.870+0000 7f651ad9d640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 0x7f650806dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f650c036000 tx=0x7f650c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:03.874+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f65140551b0 con 0x7f651c0b0fd0 2024-03-24T10:35:04.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.054+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f651c04f4e0 con 0x7f651c0b0fd0 2024-03-24T10:35:04.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.055+0000 7f65037fe640 1 -- 172.21.15.1:0/1177758026 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18362 (secure 0 0 0) 0x7f6514058870 con 0x7f651c0b0fd0 2024-03-24T10:35:04.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:04.056 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":36,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:02.917772+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 msgr2=0x7f650806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 0x7f650806dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f650c036000 tx=0x7f650c030000 comp rx=0 tx=0).stop 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 msgr2=0x7f651c0ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f651c0b5960 tx=0x7f651400f1b0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 shutdown_connections 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f650806b810 0x7f650806dcd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f651c0b0fd0 0x7f651c0ccfa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f651c1f3c70 0x7f651c0b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 --2- 172.21.15.1:0/1177758026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c1523b0 0x7f651c1f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.057+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 >> 172.21.15.1:0/1177758026 conn(0x7f651c0d3ec0 msgr2=0x7f651c0b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.058+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 shutdown_connections 2024-03-24T10:35:04.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.058+0000 7f652122e640 1 -- 172.21.15.1:0/1177758026 wait complete. 2024-03-24T10:35:04.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:04.069 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-24T10:35:04.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:04.087 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-24T10:35:04.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.295+0000 7f03fc2c3640 1 Processor -- start 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.295+0000 7f03fc2c3640 1 -- start start 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03fc2c3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f03f406d240 0x7f03f4168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03fc2c3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f40cc250 con 0x7f03f40b1380 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03fc2c3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f40d1060 con 0x7f03f40b31d0 2024-03-24T10:35:04.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03fc2c3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f03f40b35b0 con 0x7f03f406d240 2024-03-24T10:35:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03f985e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03f40b1380 0x7f03f406cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45754/0 (socket says 172.21.15.1:45754) 2024-03-24T10:35:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03f985e640 1 -- 172.21.15.1:0/2509390115 learned_addr learned my addr 172.21.15.1:0/2509390115 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:04.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112401273 0 0) 0x7f03f40cc250 con 0x7f03f40b1380 2024-03-24T10:35:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03fa860640 1 --2- 172.21.15.1:0/2509390115 >> v2:172.21.15.3:3300/0 conn(0x7f03f406d240 0x7f03f4168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.296+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e0003520 con 0x7f03f40b1380 2024-03-24T10:35:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3356781180 0 0) 0x7f03e0003520 con 0x7f03f40b1380 2024-03-24T10:35:04.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 >> v2:172.21.15.3:3300/0 conn(0x7f03f406d240 msgr2=0x7f03f4168ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 --2- 172.21.15.1:0/2509390115 >> v2:172.21.15.3:3300/0 conn(0x7f03f406d240 0x7f03f4168ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 >> v1:172.21.15.3:6789/0 conn(0x7f03f40b31d0 legacy=0x7f03f40d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f40cc4a0 con 0x7f03f40b1380 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f03e4001670 con 0x7f03f40b1380 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 >> v1:172.21.15.1:6789/0 conn(0x7f03f40b1380 legacy=0x7f03f406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03e0003970 0x7f03e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03e0005180 0x7f03e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f03f40b35b0 con 0x7f03e00047c0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f03e000c020 con 0x7f03e0005180 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f03e000c1e0 con 0x7f03e0003970 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03fa05f640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03e0003970 0x7f03e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03fa860640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03e0005180 0x7f03e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 -- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03e0003970 msgr2=0x7f03e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03e0003970 0x7f03e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 -- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03e0005180 msgr2=0x7f03e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03e0005180 0x7f03e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03f985e640 1 -- 172.21.15.1:0/2509390115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e0003520 con 0x7f03e00047c0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.297+0000 7f03fa860640 1 -- 172.21.15.1:0/2509390115 reap_dead start 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.298+0000 7f03f985e640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c210 tx=0x7f03e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a88a0b415d8ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.298+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4005ad0 con 0x7f03e00047c0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.298+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e4005c50 con 0x7f03e00047c0 2024-03-24T10:35:04.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.298+0000 7f03f905d640 1 -- 172.21.15.1:0/2509390115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e402c810 con 0x7f03e00047c0 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 -- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 msgr2=0x7f03e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c210 tx=0x7f03e00057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 -- 172.21.15.1:0/2509390115 shutdown_connections 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/2509390115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 -- 172.21.15.1:0/2509390115 >> 172.21.15.1:0/2509390115 conn(0x7f03f40d1c00 msgr2=0x7f03f40d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 -- 172.21.15.1:0/2509390115 shutdown_connections 2024-03-24T10:35:04.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.299+0000 7f03fc2c3640 1 -- 172.21.15.1:0/2509390115 wait complete. 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.300+0000 7f03fc2c3640 1 Processor -- start 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.300+0000 7f03fc2c3640 1 -- start start 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 0x7f03f40b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f41f7760 0x7f03f41f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41f7310 con 0x7f03e00047c0 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4158a50 con 0x7f03f41f7760 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fc2c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f03f41f4200 con 0x7f03f41f3d20 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35170/0 (socket says 172.21.15.1:35170) 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 -- 172.21.15.1:0/3067862857 learned_addr learned my addr 172.21.15.1:0/3067862857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03f985e640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 0x7f03f40b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa860640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f41f7760 0x7f03f41f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 -- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 msgr2=0x7f03f40b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 0x7f03f40b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 -- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f41f7760 msgr2=0x7f03f41f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f41f7760 0x7f03f41f7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 -- 172.21.15.1:0/3067862857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f41f44e0 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03f985e640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 0x7f03f40b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.301+0000 7f03fa05f640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03f000f5f0 tx=0x7f03f000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.302+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f00052e0 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.302+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f0005440 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.302+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f41f47c0 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.302+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f0003810 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.302+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41f8150 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.303+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f404f470 con 0x7f03e00047c0 2024-03-24T10:35:04.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.303+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03f0003970 con 0x7f03e00047c0 2024-03-24T10:35:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.304+0000 7f03eaffd640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.304+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f03f008a220 con 0x7f03e00047c0 2024-03-24T10:35:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.304+0000 7f03f985e640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.304+0000 7f03f985e640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03e4034000 tx=0x7f03e4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.308+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03f0054aa0 con 0x7f03e00047c0 2024-03-24T10:35:04.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.584+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7f03f40b18e0 con 0x7f03e00047c0 2024-03-24T10:35:04.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.585+0000 7f03eaffd640 1 -- 172.21.15.1:0/3067862857 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v14) ==== 210+0+167 (secure 0 0 0) 0x7f03f0058160 con 0x7f03e00047c0 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow rw" 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-24T10:35:04.586 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 msgr2=0x7f03d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03e4034000 tx=0x7f03e4009030 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 msgr2=0x7f03f40b1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03f000f5f0 tx=0x7f03f000fab0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 shutdown_connections 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03d806b810 0x7f03d806dcd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f41f7760 0x7f03f41f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03f41f3d20 0x7f03f40b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 --2- 172.21.15.1:0/3067862857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03e00047c0 0x7f03f40b1e00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 >> 172.21.15.1:0/3067862857 conn(0x7f03f40d1c00 msgr2=0x7f03f40cd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 shutdown_connections 2024-03-24T10:35:04.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.588+0000 7f03fc2c3640 1 -- 172.21.15.1:0/3067862857 wait complete. 2024-03-24T10:35:04.599 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:04.600 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:04.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:04.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.756+0000 7f809797a640 1 Processor -- start 2024-03-24T10:35:04.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.756+0000 7f809797a640 1 -- start start 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f809797a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f80900d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f809797a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f80900d1000 0x7f80900d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f809797a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f809015e770 con 0x7f80900b3320 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f809797a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f80900b3190 con 0x7f80900d1000 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f809797a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f80900d4e40 con 0x7f809015e390 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8095f17640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80900b3320 0x7f80900cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45760/0 (socket says 172.21.15.1:45760) 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8095f17640 1 -- 172.21.15.1:0/3220117764 learned_addr learned my addr 172.21.15.1:0/3220117764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8094f15640 1 --2- 172.21.15.1:0/3220117764 >> v2:172.21.15.3:3300/0 conn(0x7f80900d1000 0x7f80900d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8095716640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f80900d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2629775664 0 0) 0x7f8088001490 con 0x7f80900b3320 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f807c003520 con 0x7f80900b3320 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3605296411 0 0) 0x7f807c003520 con 0x7f80900b3320 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 msgr2=0x7f80900d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f80900d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 >> v2:172.21.15.3:3300/0 conn(0x7f80900d1000 msgr2=0x7f80900d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 --2- 172.21.15.1:0/3220117764 >> v2:172.21.15.3:3300/0 conn(0x7f80900d1000 0x7f80900d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80900d1e40 con 0x7f80900b3320 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f8088001b80 con 0x7f80900b3320 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 >> v1:172.21.15.1:6789/0 conn(0x7f80900b3320 legacy=0x7f80900cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f807c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f807c0047c0 0x7f807c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.757+0000 7f8087fff640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f807c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80900d4e40 con 0x7f807c003970 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f807c00b980 con 0x7f807c0047c0 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f807c00bb40 con 0x7f809015e390 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f807c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095f17640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f807c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 -- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 msgr2=0x7f807c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f807c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 -- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f807c0047c0 msgr2=0x7f807c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8094f15640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f807c0047c0 0x7f807c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f807c0047c0 0x7f807c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095716640 1 -- 172.21.15.1:0/3220117764 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c003520 con 0x7f807c003970 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8095f17640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f809015e390 0x7f807c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.758+0000 7f8094f15640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f807c0047c0 0x7f807c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f8095716640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f807c003d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f808c009ac0 tx=0x7f807c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2af1b24a18540b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c02c6b0 con 0x7f807c003970 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c02c810 con 0x7f807c003970 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f8087fff640 1 -- 172.21.15.1:0/3220117764 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c02cba0 con 0x7f807c003970 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f809797a640 1 -- 172.21.15.1:0/3220117764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 msgr2=0x7f807c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f809797a640 1 --2- 172.21.15.1:0/3220117764 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f807c003d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f808c009ac0 tx=0x7f807c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f8095f17640 1 -- 172.21.15.1:0/3220117764 reap_dead start 2024-03-24T10:35:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f809797a640 1 -- 172.21.15.1:0/3220117764 shutdown_connections 2024-03-24T10:35:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.759+0000 7f809797a640 1 -- 172.21.15.1:0/3220117764 >> 172.21.15.1:0/3220117764 conn(0x7f80900d3ec0 msgr2=0x7f80900d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.760+0000 7f809797a640 1 -- 172.21.15.1:0/3220117764 shutdown_connections 2024-03-24T10:35:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.760+0000 7f809797a640 1 -- 172.21.15.1:0/3220117764 wait complete. 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.761+0000 7f809797a640 1 Processor -- start 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.761+0000 7f809797a640 1 -- start start 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80900b8980 0x7f80901608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 0x7f809006bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f809009ddb0 con 0x7f807c003970 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f80900b78e0 con 0x7f8090160e10 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f80900b7bc0 con 0x7f80900b8980 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35192/0 (socket says 172.21.15.1:35192) 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 -- 172.21.15.1:0/609740862 learned_addr learned my addr 172.21.15.1:0/609740862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095f17640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 0x7f809006bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8094f15640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80900b8980 0x7f80901608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 -- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80900b8980 msgr2=0x7f80901608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80900b8980 0x7f80901608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 -- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 msgr2=0x7f809006bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 0x7f809006bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 -- 172.21.15.1:0/609740862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80900b7ea0 con 0x7f807c003970 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095f17640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 0x7f809006bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8095716640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f808c009ac0 tx=0x7f808c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c003810 con 0x7f807c003970 2024-03-24T10:35:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c003970 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.763+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80900b8180 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.764+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c003d20 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.764+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8090159d10 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.765+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809004f4e0 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.765+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f808c040450 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.765+0000 7f8085ffb640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 0x7f807406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.766+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f808c0ac4f0 con 0x7f807c003970 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.766+0000 7f8094f15640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 0x7f807406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.766+0000 7f8094f15640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 0x7f807406dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f80901613e0 tx=0x7f8080008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.770+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f808c076c50 con 0x7f807c003970 2024-03-24T10:35:04.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.951+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80900d2280 con 0x7f807c003970 2024-03-24T10:35:04.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.952+0000 7f8085ffb640 1 -- 172.21.15.1:0/609740862 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f808c07a310 con 0x7f807c003970 2024-03-24T10:35:04.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:04.953 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 msgr2=0x7f807406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 0x7f807406dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f80901613e0 tx=0x7f8080008040 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 msgr2=0x7f80901597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f808c009ac0 tx=0x7f808c004730 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 shutdown_connections 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f807406b810 0x7f807406dcd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090160e10 0x7f809006bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80900b8980 0x7f80901608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 --2- 172.21.15.1:0/609740862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f807c003970 0x7f80901597d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 >> 172.21.15.1:0/609740862 conn(0x7f80900d3ec0 msgr2=0x7f80900d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 shutdown_connections 2024-03-24T10:35:04.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:04.954+0000 7f809797a640 1 -- 172.21.15.1:0/609740862 wait complete. 2024-03-24T10:35:04.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:04.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.084+0000 7f63ce047640 1 Processor -- start 2024-03-24T10:35:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.084+0000 7f63ce047640 1 -- start start 2024-03-24T10:35:05.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63ce047640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c80d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63ce047640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c80b2ed0 0x7f63c80d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63ce047640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c80b0b70 con 0x7f63c80b1300 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63ce047640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f63c80d1d60 con 0x7f63c80b2ed0 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63ce047640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80b0de0 con 0x7f63c815e390 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63beffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c80b2ed0 0x7f63c80d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63beffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63c80b2ed0 0x7f63c80d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35196/0 (socket says 172.21.15.1:35196) 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.085+0000 7f63beffd640 1 -- 172.21.15.1:0/1765645382 learned_addr learned my addr 172.21.15.1:0/1765645382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63c77fe640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c80d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63beffd640 1 -- 172.21.15.1:0/1765645382 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 msgr2=0x7f63c80d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63beffd640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c80d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63beffd640 1 -- 172.21.15.1:0/1765645382 >> v1:172.21.15.3:6789/0 conn(0x7f63c80b1300 legacy=0x7f63c80b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63beffd640 1 -- 172.21.15.1:0/1765645382 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c815e770 con 0x7f63c80b2ed0 2024-03-24T10:35:05.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63c77fe640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c80d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.086+0000 7f63beffd640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80d2850 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd7c479bdeab296 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63c6ffd640 1 -- 172.21.15.1:0/1765645382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0030840 con 0x7f63c80b2ed0 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63c6ffd640 1 -- 172.21.15.1:0/1765645382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b00309a0 con 0x7f63c80b2ed0 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63c6ffd640 1 -- 172.21.15.1:0/1765645382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0030d30 con 0x7f63c80b2ed0 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 -- 172.21.15.1:0/1765645382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 msgr2=0x7f63c80d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80d2850 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b002b980 comp rx=0 tx=0).stop 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 -- 172.21.15.1:0/1765645382 shutdown_connections 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80d2850 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 --2- 172.21.15.1:0/1765645382 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c80d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 -- 172.21.15.1:0/1765645382 >> 172.21.15.1:0/1765645382 conn(0x7f63c80d3ee0 msgr2=0x7f63c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 -- 172.21.15.1:0/1765645382 shutdown_connections 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.087+0000 7f63ce047640 1 -- 172.21.15.1:0/1765645382 wait complete. 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 Processor -- start 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 -- start start 2024-03-24T10:35:05.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 0x7f63c80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c8160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80b9420 con 0x7f63c80b2ed0 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8159060 con 0x7f63c80b1300 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.089+0000 7f63ce047640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80b9140 con 0x7f63c815e390 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35208/0 (socket says 172.21.15.1:35208) 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 -- 172.21.15.1:0/2512143587 learned_addr learned my addr 172.21.15.1:0/2512143587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c77fe640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 0x7f63c80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c7fff640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c8160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 -- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 msgr2=0x7f63c8160ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c8160ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 -- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 msgr2=0x7f63c80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 0x7f63c80ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 -- 172.21.15.1:0/2512143587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b00094d0 con 0x7f63c80b2ed0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c77fe640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 0x7f63c80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c7fff640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c8160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63beffd640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000c00 tx=0x7f63b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0030810 con 0x7f63c80b2ed0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b0043400 con 0x7f63c80b2ed0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b00437d0 con 0x7f63c80b2ed0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c8159300 con 0x7f63c80b2ed0 2024-03-24T10:35:05.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.090+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c80b8e30 con 0x7f63c80b2ed0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.092+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c8159820 con 0x7f63c80b2ed0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.092+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f63b0043930 con 0x7f63c80b2ed0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.092+0000 7f63c4ff9640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 0x7f63a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.092+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f63b003e070 con 0x7f63c80b2ed0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.092+0000 7f63c77fe640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 0x7f63a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.093+0000 7f63c77fe640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 0x7f63a806dcd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f63c80b4d00 tx=0x7f63b8007640 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.097+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63c8159820 con 0x7f63c80b2ed0 2024-03-24T10:35:05.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.353+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f63c8159820 con 0x7f63c80b2ed0 2024-03-24T10:35:05.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.354+0000 7f63c4ff9640 1 -- 172.21.15.1:0/2512143587 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f63c8159820 con 0x7f63c80b2ed0 2024-03-24T10:35:05.354 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:05.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 msgr2=0x7f63a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 0x7f63a806dcd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f63c80b4d00 tx=0x7f63b8007640 comp rx=0 tx=0).stop 2024-03-24T10:35:05.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 msgr2=0x7f63c80b88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f63b0000c00 tx=0x7f63b0004730 comp rx=0 tx=0).stop 2024-03-24T10:35:05.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 shutdown_connections 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f63a806b810 0x7f63a806dcd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f63c815e390 0x7f63c8160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63c80b2ed0 0x7f63c80b88f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 --2- 172.21.15.1:0/2512143587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63c80b1300 0x7f63c80ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 >> 172.21.15.1:0/2512143587 conn(0x7f63c80d3ee0 msgr2=0x7f63c80cf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 shutdown_connections 2024-03-24T10:35:05.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.356+0000 7f63ce047640 1 -- 172.21.15.1:0/2512143587 wait complete. 2024-03-24T10:35:05.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.511+0000 7f0bf52c0640 1 Processor -- start 2024-03-24T10:35:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.511+0000 7f0bf52c0640 1 -- start start 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bf52c0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf01615f0 0x7f0bf00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bf52c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf00b4e40 0x7f0bf0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bf52c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf00c7a40 con 0x7f0bf01619d0 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bf52c0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf00cfd50 con 0x7f0bf01615f0 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bf52c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00b55b0 con 0x7f0bf00b4e40 2024-03-24T10:35:05.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bef7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0bf01619d0 0x7f0bf00d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45768/0 (socket says 172.21.15.1:45768) 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bef7fe640 1 -- 172.21.15.1:0/617662251 learned_addr learned my addr 172.21.15.1:0/617662251 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0beeffd640 1 --2- 172.21.15.1:0/617662251 >> v2:172.21.15.3:3300/0 conn(0x7f0bf01615f0 0x7f0bf00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0be7fff640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf00b4e40 0x7f0bf0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758173350 0 0) 0x7f0bf00c7a40 con 0x7f0bf01619d0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bdc003520 con 0x7f0bf01619d0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3467095856 0 0) 0x7f0bdc003520 con 0x7f0bf01619d0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf00b4e40 msgr2=0x7f0bf0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf00b4e40 0x7f0bf0161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 >> v2:172.21.15.3:3300/0 conn(0x7f0bf01615f0 msgr2=0x7f0bf00d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 --2- 172.21.15.1:0/617662251 >> v2:172.21.15.3:3300/0 conn(0x7f0bf01615f0 0x7f0bf00d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf00b52d0 con 0x7f0bf01619d0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.512+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f0be80019f0 con 0x7f0bf01619d0 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 >> v1:172.21.15.1:6789/0 conn(0x7f0bf01619d0 legacy=0x7f0bf00d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bdc003970 0x7f0bdc003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc0042d0 0x7f0bdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc004e10 0x7f0bdc005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00cfd50 con 0x7f0bdc0042d0 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00b55b0 con 0x7f0bdc004e10 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc00bd00 con 0x7f0bdc003970 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc0042d0 0x7f0bdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0beeffd640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bdc003970 0x7f0bdc003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0bef7fe640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc004e10 0x7f0bdc005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 -- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bdc003970 msgr2=0x7f0bdc003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bdc003970 0x7f0bdc003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 -- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc004e10 msgr2=0x7f0bdc005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc004e10 0x7f0bdc005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.513+0000 7f0be7fff640 1 -- 172.21.15.1:0/617662251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc003520 con 0x7f0bdc0042d0 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0be7fff640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc0042d0 0x7f0bdc0047c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009ac0 tx=0x7f0bdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da479899bbccde4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd802c6b0 con 0x7f0bdc0042d0 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd802c810 con 0x7f0bdc0042d0 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bee7fc640 1 -- 172.21.15.1:0/617662251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd802cba0 con 0x7f0bdc0042d0 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bf52c0640 1 -- 172.21.15.1:0/617662251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc0042d0 msgr2=0x7f0bdc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/617662251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bdc0042d0 0x7f0bdc0047c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009ac0 tx=0x7f0bdc0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.514+0000 7f0bef7fe640 1 -- 172.21.15.1:0/617662251 reap_dead start 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.515+0000 7f0bf52c0640 1 -- 172.21.15.1:0/617662251 shutdown_connections 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.515+0000 7f0bf52c0640 1 -- 172.21.15.1:0/617662251 >> 172.21.15.1:0/617662251 conn(0x7f0bf00d3ef0 msgr2=0x7f0bf00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.515+0000 7f0bf52c0640 1 -- 172.21.15.1:0/617662251 shutdown_connections 2024-03-24T10:35:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.515+0000 7f0bf52c0640 1 -- 172.21.15.1:0/617662251 wait complete. 2024-03-24T10:35:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 Processor -- start 2024-03-24T10:35:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 -- start start 2024-03-24T10:35:05.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc0042d0 0x7f0bf00b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf0155530 0x7f0bf00cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0157330 con 0x7f0bf0156940 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00b7f80 con 0x7f0bdc0042d0 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.516+0000 7f0bf52c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00b7140 con 0x7f0bf0155530 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35222/0 (socket says 172.21.15.1:35222) 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 -- 172.21.15.1:0/1328242823 learned_addr learned my addr 172.21.15.1:0/1328242823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0beeffd640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc0042d0 0x7f0bf00b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0be7fff640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf0155530 0x7f0bf00cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 -- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf0155530 msgr2=0x7f0bf00cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf0155530 0x7f0bf00cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 -- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc0042d0 msgr2=0x7f0bf00b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc0042d0 0x7f0bf00b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 -- 172.21.15.1:0/1328242823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf00b7420 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bef7fe640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00b9fa0 tx=0x7f0be8007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be800ab60 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be800acc0 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf00b7700 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8013630 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.517+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf00ce0c0 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.518+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf004f470 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.519+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0be8013790 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.520+0000 7f0be77fe640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 0x7f0bd006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.520+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f0be808a960 con 0x7f0bf0156940 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.520+0000 7f0beeffd640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 0x7f0bd006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.520+0000 7f0beeffd640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 0x7f0bd006dd00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009c40 tx=0x7f0be0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.524+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0be80551f0 con 0x7f0bf0156940 2024-03-24T10:35:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.704+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0bf00b80c0 con 0x7f0bf0156940 2024-03-24T10:35:05.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.704+0000 7f0be77fe640 1 -- 172.21.15.1:0/1328242823 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f0be80588b0 con 0x7f0bf0156940 2024-03-24T10:35:05.705 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 msgr2=0x7f0bd006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 0x7f0bd006dd00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009c40 tx=0x7f0be0009190 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 msgr2=0x7f0bf0156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00b9fa0 tx=0x7f0be8007b10 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 shutdown_connections 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0bd006b840 0x7f0bd006dd00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bf0156940 0x7f0bf0156df0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0bf0155530 0x7f0bf00cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 --2- 172.21.15.1:0/1328242823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bdc0042d0 0x7f0bf00b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 >> 172.21.15.1:0/1328242823 conn(0x7f0bf00d3ef0 msgr2=0x7f0bf00d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 shutdown_connections 2024-03-24T10:35:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.706+0000 7f0bf52c0640 1 -- 172.21.15.1:0/1328242823 wait complete. 2024-03-24T10:35:05.719 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.834+0000 7f9880d63640 1 Processor -- start 2024-03-24T10:35:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.834+0000 7f9880d63640 1 -- start start 2024-03-24T10:35:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9880d63640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f987c0cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9880d63640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c0b0b70 con 0x7f987c0d2a20 2024-03-24T10:35:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9880d63640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c0d1d60 con 0x7f987c15e390 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9880d63640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f987c0b0de0 con 0x7f987c0b2310 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9879d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f987c0d2a20 0x7f987c0cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45778/0 (socket says 172.21.15.1:45778) 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.835+0000 7f9879d9b640 1 -- 172.21.15.1:0/2278306662 learned_addr learned my addr 172.21.15.1:0/2278306662 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987ad9d640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f987c0cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 531199793 0 0) 0x7f987c0b0b70 con 0x7f987c0d2a20 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9864003520 con 0x7f987c0d2a20 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139512254 0 0) 0x7f987c0d1d60 con 0x7f987c15e390 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c0b0b70 con 0x7f987c15e390 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4050631863 0 0) 0x7f9864003520 con 0x7f987c0d2a20 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 msgr2=0x7f987c0cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f987c0cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 >> v1:172.21.15.3:6789/0 conn(0x7f987c15e390 legacy=0x7f987c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c15e770 con 0x7f987c0d2a20 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f986c001670 con 0x7f987c0d2a20 2024-03-24T10:35:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987ad9d640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f987c0cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 >> v1:172.21.15.1:6789/0 conn(0x7f987c0d2a20 legacy=0x7f987c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f9864003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9864004070 0x7f98640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004d00 0x7f9864005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f987c0b0de0 con 0x7f9864004070 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f986400ba30 con 0x7f9864004d00 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f986400bbf0 con 0x7f987c0b2310 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f9879d9b640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9864004070 0x7f98640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987ad9d640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004d00 0x7f9864005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.836+0000 7f987a59c640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f9864003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 -- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 msgr2=0x7f9864003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f9864003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 -- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004d00 msgr2=0x7f9864005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004d00 0x7f9864005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 -- 172.21.15.1:0/2278306662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9874009a70 con 0x7f9864004070 2024-03-24T10:35:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f987a59c640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c0b2310 0x7f9864003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f9879d9b640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9864004070 0x7f98640047c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f986c007fd0 tx=0x7f98640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da3b16c951dd9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.837+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c005ad0 con 0x7f9864004070 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986c005c50 con 0x7f9864004070 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f987959a640 1 -- 172.21.15.1:0/2278306662 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c02c830 con 0x7f9864004070 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 -- 172.21.15.1:0/2278306662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9864004070 msgr2=0x7f98640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 --2- 172.21.15.1:0/2278306662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9864004070 0x7f98640047c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f986c007fd0 tx=0x7f98640057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f987ad9d640 1 -- 172.21.15.1:0/2278306662 reap_dead start 2024-03-24T10:35:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 -- 172.21.15.1:0/2278306662 shutdown_connections 2024-03-24T10:35:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 -- 172.21.15.1:0/2278306662 >> 172.21.15.1:0/2278306662 conn(0x7f987c0d3ee0 msgr2=0x7f987c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 -- 172.21.15.1:0/2278306662 shutdown_connections 2024-03-24T10:35:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.838+0000 7f9880d63640 1 -- 172.21.15.1:0/2278306662 wait complete. 2024-03-24T10:35:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.839+0000 7f9880d63640 1 Processor -- start 2024-03-24T10:35:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 -- start start 2024-03-24T10:35:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004070 0x7f987c0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c1606d0 0x7f987c0cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f987c15f310 con 0x7f987c160b30 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f987c15f520 con 0x7f9864004070 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f9880d63640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f987c15f800 con 0x7f987c1606d0 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35244/0 (socket says 172.21.15.1:35244) 2024-03-24T10:35:05.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.840+0000 7f987ad9d640 1 -- 172.21.15.1:0/3354605935 learned_addr learned my addr 172.21.15.1:0/3354605935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987a59c640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004070 0x7f987c0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f9879d9b640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c1606d0 0x7f987c0cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 -- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c1606d0 msgr2=0x7f987c0cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c1606d0 0x7f987c0cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 -- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004070 msgr2=0x7f987c0cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004070 0x7f987c0cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 -- 172.21.15.1:0/3354605935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c15fae0 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f987ad9d640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f987c0d3940 tx=0x7f987400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987400e040 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987400f040 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c15fdc0 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874015670 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.841+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c0cf570 con 0x7f987c160b30 2024-03-24T10:35:05.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.842+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c0cc000 con 0x7f987c160b30 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.842+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f98740157d0 con 0x7f987c160b30 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.843+0000 7f986affd640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 0x7f985c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.843+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f987408ab60 con 0x7f987c160b30 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.843+0000 7f987a59c640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 0x7f985c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.843+0000 7f987a59c640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 0x7f985c06dd00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9870000f60 tx=0x7f9870009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:05.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:05.851+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f987c0cc000 con 0x7f987c160b30 2024-03-24T10:35:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.053+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f987c0cc000 con 0x7f987c160b30 2024-03-24T10:35:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.054+0000 7f986affd640 1 -- 172.21.15.1:0/3354605935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7f987c0cc000 con 0x7f987c160b30 2024-03-24T10:35:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 msgr2=0x7f985c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 0x7f985c06dd00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9870000f60 tx=0x7f9870009040 comp rx=0 tx=0).stop 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 msgr2=0x7f987c0cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f987c0d3940 tx=0x7f987400f580 comp rx=0 tx=0).stop 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 shutdown_connections 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f985c06b840 0x7f985c06dd00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c160b30 0x7f987c0cc820 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f987c1606d0 0x7f987c0cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 --2- 172.21.15.1:0/3354605935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9864004070 0x7f987c0cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 >> 172.21.15.1:0/3354605935 conn(0x7f987c0d3ee0 msgr2=0x7f987c0c7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 shutdown_connections 2024-03-24T10:35:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.057+0000 7f9880d63640 1 -- 172.21.15.1:0/3354605935 wait complete. 2024-03-24T10:35:06.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 Processor -- start 2024-03-24T10:35:06.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 -- start start 2024-03-24T10:35:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 0x7fbc3c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbc3c0b4e40 0x7fbc3c161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3c0c7a40 con 0x7fbc3c1619d0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbc3c0cfd50 con 0x7fbc3c0b4e40 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.174+0000 7fbc408d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3c0b55b0 con 0x7fbc3c1615f0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc3ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbc3c1619d0 0x7fbc3c0d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45788/0 (socket says 172.21.15.1:45788) 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc3ad9d640 1 -- 172.21.15.1:0/392856188 learned_addr learned my addr 172.21.15.1:0/392856188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc33fff640 1 --2- 172.21.15.1:0/392856188 >> v2:172.21.15.3:3300/0 conn(0x7fbc3c0b4e40 0x7fbc3c161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc3a59c640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 0x7fbc3c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc39d9b640 1 -- 172.21.15.1:0/392856188 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931387530 0 0) 0x7fbc3c0c7a40 con 0x7fbc3c1619d0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc39d9b640 1 -- 172.21.15.1:0/392856188 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc28003520 con 0x7fbc3c1619d0 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc33fff640 1 -- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 msgr2=0x7fbc3c0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc33fff640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 0x7fbc3c0d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc33fff640 1 -- 172.21.15.1:0/392856188 >> v1:172.21.15.1:6789/0 conn(0x7fbc3c1619d0 legacy=0x7fbc3c0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc33fff640 1 -- 172.21.15.1:0/392856188 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3c0b52d0 con 0x7fbc3c0b4e40 2024-03-24T10:35:06.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.175+0000 7fbc3a59c640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 0x7fbc3c0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.176+0000 7fbc33fff640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c0b4e40 0x7fbc3c161f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e91f5e7e7082441 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.176+0000 7fbc39d9b640 1 -- 172.21.15.1:0/392856188 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24031830 con 0x7fbc3c0b4e40 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.176+0000 7fbc39d9b640 1 -- 172.21.15.1:0/392856188 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc24031990 con 0x7fbc3c0b4e40 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.176+0000 7fbc39d9b640 1 -- 172.21.15.1:0/392856188 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24031d20 con 0x7fbc3c0b4e40 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 -- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c0b4e40 msgr2=0x7fbc3c161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c0b4e40 0x7fbc3c161f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 -- 172.21.15.1:0/392856188 shutdown_connections 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c0b4e40 0x7fbc3c161f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 --2- 172.21.15.1:0/392856188 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1615f0 0x7fbc3c0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 -- 172.21.15.1:0/392856188 >> 172.21.15.1:0/392856188 conn(0x7fbc3c0d3ef0 msgr2=0x7fbc3c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 -- 172.21.15.1:0/392856188 shutdown_connections 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.177+0000 7fbc408d4640 1 -- 172.21.15.1:0/392856188 wait complete. 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.178+0000 7fbc408d4640 1 Processor -- start 2024-03-24T10:35:06.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.178+0000 7fbc408d4640 1 -- start start 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c1615f0 0x7fbc3c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1619d0 0x7fbc3c0b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c0b8480 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c150120 con 0x7fbc3c1615f0 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc408d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3c0cb7b0 con 0x7fbc3c1619d0 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35260/0 (socket says 172.21.15.1:35260) 2024-03-24T10:35:06.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 -- 172.21.15.1:0/3719497383 learned_addr learned my addr 172.21.15.1:0/3719497383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc33fff640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1619d0 0x7fbc3c0b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3a59c640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c1615f0 0x7fbc3c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 -- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1619d0 msgr2=0x7fbc3c0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1619d0 0x7fbc3c0b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 -- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c1615f0 msgr2=0x7fbc3c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c1615f0 0x7fbc3c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 -- 172.21.15.1:0/3719497383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c009400 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.179+0000 7fbc3ad9d640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400cad0 tx=0x7fbc3400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.180+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3400ac30 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.180+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc3400ad90 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.180+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc24009400 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.180+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34012620 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.180+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3c0cb160 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.181+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc3c0cafc0 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.181+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc34007070 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.182+0000 7fbc32ffd640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 0x7fbc1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.182+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fbc3408a360 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.182+0000 7fbc3a59c640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 0x7fbc1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.182+0000 7fbc3a59c640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 0x7fbc1c06dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009cd0 tx=0x7fbc2c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.190+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc3c0cafc0 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.369+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fbc3c0cafc0 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.369+0000 7fbc32ffd640 1 -- 172.21.15.1:0/3719497383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7fbc3c0cafc0 con 0x7fbc3c1f7f50 2024-03-24T10:35:06.369 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 msgr2=0x7fbc1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 0x7fbc1c06dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009cd0 tx=0x7fbc2c009190 comp rx=0 tx=0).stop 2024-03-24T10:35:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 msgr2=0x7fbc3c0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400cad0 tx=0x7fbc3400cf90 comp rx=0 tx=0).stop 2024-03-24T10:35:06.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 shutdown_connections 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbc1c06b810 0x7fbc1c06dcd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc3c1f7f50 0x7fbc3c0cd7a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbc3c1619d0 0x7fbc3c0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 --2- 172.21.15.1:0/3719497383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc3c1615f0 0x7fbc3c0cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 >> 172.21.15.1:0/3719497383 conn(0x7fbc3c0d3ef0 msgr2=0x7fbc3c0c9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 shutdown_connections 2024-03-24T10:35:06.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.371+0000 7fbc408d4640 1 -- 172.21.15.1:0/3719497383 wait complete. 2024-03-24T10:35:06.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.488+0000 7fa64aaf1640 1 Processor -- start 2024-03-24T10:35:06.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.488+0000 7fa64aaf1640 1 -- start start 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64aaf1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6440d2bb0 0x7fa6440b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64aaf1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64415e770 con 0x7fa6440d1000 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64aaf1640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6440d4e40 con 0x7fa64415e390 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64aaf1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa6440b3190 con 0x7fa6440d2bb0 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa643fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6440d2bb0 0x7fa6440b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64908e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa6440d1000 0x7fa6440b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45790/0 (socket says 172.21.15.1:45790) 2024-03-24T10:35:06.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa64908e640 1 -- 172.21.15.1:0/2054179084 learned_addr learned my addr 172.21.15.1:0/2054179084 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa643fff640 1 -- 172.21.15.1:0/2054179084 >> v1:172.21.15.3:6789/0 conn(0x7fa64415e390 legacy=0x7fa6440d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa643fff640 1 -- 172.21.15.1:0/2054179084 >> v1:172.21.15.1:6789/0 conn(0x7fa6440d1000 legacy=0x7fa6440b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.489+0000 7fa643fff640 1 -- 172.21.15.1:0/2054179084 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6440d1e40 con 0x7fa6440d2bb0 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.490+0000 7fa643fff640 1 --2- 172.21.15.1:0/2054179084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6440d2bb0 0x7fa6440b3b90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6300097a0 tx=0x7fa63002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8f1e26e9cd092b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.490+0000 7fa6437fe640 1 -- 172.21.15.1:0/2054179084 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63002e070 con 0x7fa6440d2bb0 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.490+0000 7fa6437fe640 1 -- 172.21.15.1:0/2054179084 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6300315a0 con 0x7fa6440d2bb0 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.490+0000 7fa6437fe640 1 -- 172.21.15.1:0/2054179084 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630031930 con 0x7fa6440d2bb0 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 -- 172.21.15.1:0/2054179084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6440d2bb0 msgr2=0x7fa6440b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/2054179084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6440d2bb0 0x7fa6440b3b90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6300097a0 tx=0x7fa63002b840 comp rx=0 tx=0).stop 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 -- 172.21.15.1:0/2054179084 shutdown_connections 2024-03-24T10:35:06.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/2054179084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6440d2bb0 0x7fa6440b3b90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 -- 172.21.15.1:0/2054179084 >> 172.21.15.1:0/2054179084 conn(0x7fa6440d3ec0 msgr2=0x7fa6440d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 -- 172.21.15.1:0/2054179084 shutdown_connections 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.491+0000 7fa64aaf1640 1 -- 172.21.15.1:0/2054179084 wait complete. 2024-03-24T10:35:06.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.492+0000 7fa64aaf1640 1 Processor -- start 2024-03-24T10:35:06.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.492+0000 7fa64aaf1640 1 -- start start 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6440d1000 0x7fa64406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa6440d2bb0 0x7fa6440cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6440a04b0 con 0x7fa64415e390 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa64409e910 con 0x7fa6440d1000 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64aaf1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa64409dd40 con 0x7fa6440d2bb0 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35292/0 (socket says 172.21.15.1:35292) 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 -- 172.21.15.1:0/216697378 learned_addr learned my addr 172.21.15.1:0/216697378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa643fff640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa6440d2bb0 0x7fa6440cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64888d640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6440d1000 0x7fa64406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 -- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa6440d2bb0 msgr2=0x7fa6440cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa6440d2bb0 0x7fa6440cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 -- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6440d1000 msgr2=0x7fa64406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6440d1000 0x7fa64406bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 -- 172.21.15.1:0/216697378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa630009400 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.493+0000 7fa64908e640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00cbc0 tx=0x7fa63c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.494+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00ad40 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.494+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63c012400 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.494+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00aea0 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.494+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64409dfe0 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.494+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64409e5a0 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.495+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64404f4e0 con 0x7fa64415e390 2024-03-24T10:35:06.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.495+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa63c007070 con 0x7fa64415e390 2024-03-24T10:35:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.496+0000 7fa6417fa640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 0x7fa62806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.496+0000 7fa64888d640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 0x7fa62806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.496+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fa63c08a030 con 0x7fa64415e390 2024-03-24T10:35:06.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.496+0000 7fa64888d640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 0x7fa62806dcd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa6380016e0 tx=0x7fa638009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.500+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa63c054a00 con 0x7fa64415e390 2024-03-24T10:35:06.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.694+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa64415e770 con 0x7fa64415e390 2024-03-24T10:35:06.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.695+0000 7fa6417fa640 1 -- 172.21.15.1:0/216697378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7fa64415e770 con 0x7fa64415e390 2024-03-24T10:35:06.695 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 msgr2=0x7fa62806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 0x7fa62806dcd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa6380016e0 tx=0x7fa638009040 comp rx=0 tx=0).stop 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 msgr2=0x7fa6440cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00cbc0 tx=0x7fa63c00a430 comp rx=0 tx=0).stop 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 shutdown_connections 2024-03-24T10:35:06.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa62806b810 0x7fa62806dcd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64415e390 0x7fa6440cde20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa6440d2bb0 0x7fa6440cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 --2- 172.21.15.1:0/216697378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6440d1000 0x7fa64406bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 >> 172.21.15.1:0/216697378 conn(0x7fa6440d3ec0 msgr2=0x7fa6440d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 shutdown_connections 2024-03-24T10:35:06.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.697+0000 7fa64aaf1640 1 -- 172.21.15.1:0/216697378 wait complete. 2024-03-24T10:35:06.711 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:06.711 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:06.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.868+0000 7f830ee07640 1 Processor -- start 2024-03-24T10:35:06.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.868+0000 7f830ee07640 1 -- start start 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830ee07640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83080d2bb0 0x7f83080b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830ee07640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f830815e770 con 0x7f830815e390 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830ee07640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83080d4e40 con 0x7f83080d1000 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830ee07640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f83080b3190 con 0x7f83080d2bb0 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f8307fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83080d2bb0 0x7f83080b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830cba3640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f830815e390 0x7f83080d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45804/0 (socket says 172.21.15.1:45804) 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f830cba3640 1 -- 172.21.15.1:0/1586926169 learned_addr learned my addr 172.21.15.1:0/1586926169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f8307fff640 1 -- 172.21.15.1:0/1586926169 >> v1:172.21.15.3:6789/0 conn(0x7f83080d1000 legacy=0x7f83080b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.869+0000 7f8307fff640 1 -- 172.21.15.1:0/1586926169 >> v1:172.21.15.1:6789/0 conn(0x7f830815e390 legacy=0x7f83080d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f8307fff640 1 -- 172.21.15.1:0/1586926169 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83080d1e40 con 0x7f83080d2bb0 2024-03-24T10:35:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f83077fe640 1 -- 172.21.15.1:0/1586926169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830637067 0 0) 0x7f82fc001110 con 0x7f830815e390 2024-03-24T10:35:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f8307fff640 1 --2- 172.21.15.1:0/1586926169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d2bb0 0x7f83080b3b90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f82f40097a0 tx=0x7f82f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35834c7da851380 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f83077fe640 1 -- 172.21.15.1:0/1586926169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f402e070 con 0x7f83080d2bb0 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f83077fe640 1 -- 172.21.15.1:0/1586926169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f40315a0 con 0x7f83080d2bb0 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.870+0000 7f83077fe640 1 -- 172.21.15.1:0/1586926169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4031930 con 0x7f83080d2bb0 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 -- 172.21.15.1:0/1586926169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d2bb0 msgr2=0x7f83080b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 --2- 172.21.15.1:0/1586926169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d2bb0 0x7f83080b3b90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f82f40097a0 tx=0x7f82f402b840 comp rx=0 tx=0).stop 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 -- 172.21.15.1:0/1586926169 shutdown_connections 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 --2- 172.21.15.1:0/1586926169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d2bb0 0x7f83080b3b90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 -- 172.21.15.1:0/1586926169 >> 172.21.15.1:0/1586926169 conn(0x7f83080d3ec0 msgr2=0x7f83080d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 -- 172.21.15.1:0/1586926169 shutdown_connections 2024-03-24T10:35:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.871+0000 7f830ee07640 1 -- 172.21.15.1:0/1586926169 wait complete. 2024-03-24T10:35:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.872+0000 7f830ee07640 1 Processor -- start 2024-03-24T10:35:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 -- start start 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 0x7f830809ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f830815e390 0x7f83080ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8308155920 con 0x7f83080d1000 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f830809f290 con 0x7f83080d2bb0 2024-03-24T10:35:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830ee07640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f830809dcb0 con 0x7f830815e390 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830cba3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830cba3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35310/0 (socket says 172.21.15.1:35310) 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.873+0000 7f830cba3640 1 -- 172.21.15.1:0/3684575362 learned_addr learned my addr 172.21.15.1:0/3684575362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f8307fff640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 0x7f830809ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 -- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f830815e390 msgr2=0x7f83080ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830d3a4640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f830815e390 0x7f83080ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f830815e390 0x7f83080ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 -- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 msgr2=0x7f830809ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 0x7f830809ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 -- 172.21.15.1:0/3684575362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f4009400 con 0x7f83080d1000 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f8307fff640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 0x7f830809ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830cba3640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c3f0 tx=0x7f82fc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc0074e0 con 0x7f83080d1000 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82fc007640 con 0x7f83080d1000 2024-03-24T10:35:06.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830809df90 con 0x7f83080d1000 2024-03-24T10:35:06.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc004a00 con 0x7f83080d1000 2024-03-24T10:35:06.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.874+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830809e550 con 0x7f83080d1000 2024-03-24T10:35:06.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.875+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830804f4e0 con 0x7f83080d1000 2024-03-24T10:35:06.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.876+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82fc004b60 con 0x7f83080d1000 2024-03-24T10:35:06.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.876+0000 7f83057fa640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 0x7f82ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:06.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.876+0000 7f8307fff640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 0x7f82ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:06.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.876+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f82fc089f50 con 0x7f83080d1000 2024-03-24T10:35:06.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.877+0000 7f8307fff640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 0x7f82ec06dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f82f4036000 tx=0x7f82f403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:06.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:06.880+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82fc0547e0 con 0x7f83080d1000 2024-03-24T10:35:07.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.189+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f830815e770 con 0x7f83080d1000 2024-03-24T10:35:07.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.190+0000 7f83057fa640 1 -- 172.21.15.1:0/3684575362 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f830815e770 con 0x7f83080d1000 2024-03-24T10:35:07.230 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:07.230 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:07.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 msgr2=0x7f82ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 0x7f82ec06dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f82f4036000 tx=0x7f82f403d040 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 msgr2=0x7f83081553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00c3f0 tx=0x7f82fc00c8b0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 shutdown_connections 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f82ec06b810 0x7f82ec06dcd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f830815e390 0x7f83080ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83080d2bb0 0x7f830809ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 --2- 172.21.15.1:0/3684575362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83080d1000 0x7f83081553e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 >> 172.21.15.1:0/3684575362 conn(0x7f83080d3ec0 msgr2=0x7f830806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 shutdown_connections 2024-03-24T10:35:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.193+0000 7f830ee07640 1 -- 172.21.15.1:0/3684575362 wait complete. 2024-03-24T10:35:07.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:07.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.337+0000 7f3963a6f640 1 Processor -- start 2024-03-24T10:35:07.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.337+0000 7f3963a6f640 1 -- start start 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f3963a6f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f395c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f3963a6f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f395c0b0b70 con 0x7f395c0b2310 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f3963a6f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f395c0d1d60 con 0x7f395c15e390 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f3963a6f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f395c0b0de0 con 0x7f395c0d2a20 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f396200c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f395c0b2310 0x7f395c0cf010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45818/0 (socket says 172.21.15.1:45818) 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f396200c640 1 -- 172.21.15.1:0/1788995405 learned_addr learned my addr 172.21.15.1:0/1788995405 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f396100a640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f395c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.338+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840683654 0 0) 0x7f395c0b0b70 con 0x7f395c0b2310 2024-03-24T10:35:07.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3948003520 con 0x7f395c0b2310 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2320426757 0 0) 0x7f3948003520 con 0x7f395c0b2310 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 msgr2=0x7f395c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f395c0cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 >> v1:172.21.15.3:6789/0 conn(0x7f395c15e390 legacy=0x7f395c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c15e770 con 0x7f395c0b2310 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f39540019f0 con 0x7f395c0b2310 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 >> v1:172.21.15.1:6789/0 conn(0x7f395c0b2310 legacy=0x7f395c0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39480039a0 0x7f3948003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f39480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3948004d00 0x7f3948005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f395c0b0de0 con 0x7f39480039a0 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f394800b980 con 0x7f3948004d00 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f394800bb40 con 0x7f395c0d2a20 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39480039a0 0x7f3948003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396100a640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f39480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396200c640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3948004d00 0x7f3948005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 -- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 msgr2=0x7f39480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f39480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 -- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3948004d00 msgr2=0x7f3948005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3948004d00 0x7f3948005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396180b640 1 -- 172.21.15.1:0/1788995405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3948003520 con 0x7f39480039a0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.339+0000 7f396100a640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0d2a20 0x7f39480047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.340+0000 7f396180b640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39480039a0 0x7f3948003d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3958001860 tx=0x7f39480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332998fe1350eb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.340+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958033640 con 0x7f39480039a0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.340+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39580337c0 con 0x7f39480039a0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.340+0000 7f3960809640 1 -- 172.21.15.1:0/1788995405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958033b70 con 0x7f39480039a0 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 -- 172.21.15.1:0/1788995405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39480039a0 msgr2=0x7f3948003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 --2- 172.21.15.1:0/1788995405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39480039a0 0x7f3948003d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3958001860 tx=0x7f39480057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f396200c640 1 -- 172.21.15.1:0/1788995405 reap_dead start 2024-03-24T10:35:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 -- 172.21.15.1:0/1788995405 shutdown_connections 2024-03-24T10:35:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 -- 172.21.15.1:0/1788995405 >> 172.21.15.1:0/1788995405 conn(0x7f395c0d3ee0 msgr2=0x7f395c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 -- 172.21.15.1:0/1788995405 shutdown_connections 2024-03-24T10:35:07.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.341+0000 7f3963a6f640 1 -- 172.21.15.1:0/1788995405 wait complete. 2024-03-24T10:35:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.342+0000 7f3963a6f640 1 Processor -- start 2024-03-24T10:35:07.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.342+0000 7f3963a6f640 1 -- start start 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39480039a0 0x7f395c152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0b7820 0x7f395c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f395c0cd860 con 0x7f395c0cd3d0 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f395c0cbf30 con 0x7f39480039a0 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f3963a6f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f395c0cc210 con 0x7f395c0b7820 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35326/0 (socket says 172.21.15.1:35326) 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 -- 172.21.15.1:0/3912681816 learned_addr learned my addr 172.21.15.1:0/3912681816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396180b640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39480039a0 0x7f395c152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396100a640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0b7820 0x7f395c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 -- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0b7820 msgr2=0x7f395c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0b7820 0x7f395c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 -- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39480039a0 msgr2=0x7f395c152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39480039a0 0x7f395c152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 -- 172.21.15.1:0/3912681816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c15e9a0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f396200c640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f395c0d3940 tx=0x7f3954007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.343+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395400ac20 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.344+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f395400ada0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.344+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395c15ec80 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.344+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954013630 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.344+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395c0cb6f0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.345+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395c15f1a0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.345+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3954013790 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.346+0000 7f394e7fc640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 0x7f394006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.346+0000 7f396180b640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 0x7f394006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.346+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f395408ac80 con 0x7f395c0cd3d0 2024-03-24T10:35:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.346+0000 7f396180b640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 0x7f394006dd00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3958035000 tx=0x7f3958009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.350+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f395c15f1a0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.533+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f395c15f1a0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.533+0000 7f394e7fc640 1 -- 172.21.15.1:0/3912681816 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f395c15f1a0 con 0x7f395c0cd3d0 2024-03-24T10:35:07.533 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 msgr2=0x7f394006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 0x7f394006dd00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3958035000 tx=0x7f3958009030 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 msgr2=0x7f395c0cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f395c0d3940 tx=0x7f3954007bd0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 shutdown_connections 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f394006b840 0x7f394006dd00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f395c0cd3d0 0x7f395c0cb1b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f395c0b7820 0x7f395c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 --2- 172.21.15.1:0/3912681816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39480039a0 0x7f395c152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 >> 172.21.15.1:0/3912681816 conn(0x7f395c0d3ee0 msgr2=0x7f395c0cfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 shutdown_connections 2024-03-24T10:35:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.535+0000 7f3963a6f640 1 -- 172.21.15.1:0/3912681816 wait complete. 2024-03-24T10:35:07.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.650+0000 7f4e5c53e640 1 Processor -- start 2024-03-24T10:35:07.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.651+0000 7f4e5c53e640 1 -- start start 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.651+0000 7f4e5c53e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e540b4e40 0x7f4e540d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.651+0000 7f4e5c53e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e540c7a40 con 0x7f4e54162610 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.651+0000 7f4e5c53e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e540b55b0 con 0x7f4e541615f0 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.651+0000 7f4e5c53e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4e540cfd50 con 0x7f4e540b4e40 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e5aadb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4e54162610 0x7f4e540d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45824/0 (socket says 172.21.15.1:45824) 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e5aadb640 1 -- 172.21.15.1:0/71453638 learned_addr learned my addr 172.21.15.1:0/71453638 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e59ad9640 1 --2- 172.21.15.1:0/71453638 >> v2:172.21.15.3:3300/0 conn(0x7f4e540b4e40 0x7f4e540d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471828292 0 0) 0x7f4e540c7a40 con 0x7f4e54162610 2024-03-24T10:35:07.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e40003520 con 0x7f4e54162610 2024-03-24T10:35:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2889049076 0 0) 0x7f4e40003520 con 0x7f4e54162610 2024-03-24T10:35:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e59ad9640 1 -- 172.21.15.1:0/71453638 >> v1:172.21.15.3:6789/0 conn(0x7f4e541615f0 legacy=0x7f4e54161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e59ad9640 1 -- 172.21.15.1:0/71453638 >> v1:172.21.15.1:6789/0 conn(0x7f4e54162610 legacy=0x7f4e540d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.652+0000 7f4e59ad9640 1 -- 172.21.15.1:0/71453638 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e540b52d0 con 0x7f4e540b4e40 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.653+0000 7f4e59ad9640 1 --2- 172.21.15.1:0/71453638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b4e40 0x7f4e540d4a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4e440097a0 tx=0x7f4e4402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b173400b452ef53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.653+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4402f050 con 0x7f4e540b4e40 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.653+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e44031500 con 0x7f4e540b4e40 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.653+0000 7f4e592d8640 1 -- 172.21.15.1:0/71453638 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0019f0 con 0x7f4e540b4e40 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 -- 172.21.15.1:0/71453638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b4e40 msgr2=0x7f4e540d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/71453638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b4e40 0x7f4e540d4a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4e440097a0 tx=0x7f4e4402b820 comp rx=0 tx=0).stop 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 -- 172.21.15.1:0/71453638 shutdown_connections 2024-03-24T10:35:07.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/71453638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b4e40 0x7f4e540d4a10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 -- 172.21.15.1:0/71453638 >> 172.21.15.1:0/71453638 conn(0x7f4e540d3ef0 msgr2=0x7f4e540d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 -- 172.21.15.1:0/71453638 shutdown_connections 2024-03-24T10:35:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.654+0000 7f4e5c53e640 1 -- 172.21.15.1:0/71453638 wait complete. 2024-03-24T10:35:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.655+0000 7f4e5c53e640 1 Processor -- start 2024-03-24T10:35:07.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.655+0000 7f4e5c53e640 1 -- start start 2024-03-24T10:35:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e541615f0 0x7f4e54158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b9410 0x7f4e540c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4e541f8610 con 0x7f4e54162610 2024-03-24T10:35:07.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4e541595e0 con 0x7f4e540b9410 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4e541589d0 con 0x7f4e541615f0 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35340/0 (socket says 172.21.15.1:35340) 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 -- 172.21.15.1:0/3997686576 learned_addr learned my addr 172.21.15.1:0/3997686576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5aadb640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b9410 0x7f4e540c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5a2da640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e541615f0 0x7f4e54158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 -- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e541615f0 msgr2=0x7f4e54158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e541615f0 0x7f4e54158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 -- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b9410 msgr2=0x7f4e540c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b9410 0x7f4e540c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 -- 172.21.15.1:0/3997686576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e44009400 con 0x7f4e54162610 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e59ad9640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402b860 tx=0x7f4e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44033980 con 0x7f4e54162610 2024-03-24T10:35:07.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e44033ae0 con 0x7f4e54162610 2024-03-24T10:35:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e54158cb0 con 0x7f4e54162610 2024-03-24T10:35:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.656+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44033e90 con 0x7f4e54162610 2024-03-24T10:35:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.657+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e54159270 con 0x7f4e54162610 2024-03-24T10:35:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.657+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5404f470 con 0x7f4e54162610 2024-03-24T10:35:07.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.658+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e4403a040 con 0x7f4e54162610 2024-03-24T10:35:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.658+0000 7f4e4affd640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 0x7f4e3806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.658+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f4e440abc30 con 0x7f4e54162610 2024-03-24T10:35:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.658+0000 7f4e5a2da640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 0x7f4e3806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.659+0000 7f4e5a2da640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 0x7f4e3806dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4e541eb130 tx=0x7f4e50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.663+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e44076570 con 0x7f4e54162610 2024-03-24T10:35:07.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.843+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e54159720 con 0x7f4e54162610 2024-03-24T10:35:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.844+0000 7f4e4affd640 1 -- 172.21.15.1:0/3997686576 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f4e44079c30 con 0x7f4e54162610 2024-03-24T10:35:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 msgr2=0x7f4e3806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 0x7f4e3806dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4e541eb130 tx=0x7f4e50009040 comp rx=0 tx=0).stop 2024-03-24T10:35:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 msgr2=0x7f4e541f80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402b860 tx=0x7f4e440046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 shutdown_connections 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4e3806b840 0x7f4e3806dd00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e540b9410 0x7f4e540c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e54162610 0x7f4e541f80d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 --2- 172.21.15.1:0/3997686576 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4e541615f0 0x7f4e54158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 >> 172.21.15.1:0/3997686576 conn(0x7f4e540d3ef0 msgr2=0x7f4e5406c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 shutdown_connections 2024-03-24T10:35:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.846+0000 7f4e5c53e640 1 -- 172.21.15.1:0/3997686576 wait complete. 2024-03-24T10:35:07.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.963+0000 7efcbf447640 1 Processor -- start 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.963+0000 7efcbf447640 1 -- start start 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbf447640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efcb815e390 0x7efcb80d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbf447640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efcb80b2ed0 0x7efcb80d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbf447640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb80b0b70 con 0x7efcb80b1300 2024-03-24T10:35:07.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbf447640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efcb80d1d60 con 0x7efcb80b2ed0 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbf447640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7efcb80b0de0 con 0x7efcb815e390 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbc9e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efcb80b2ed0 0x7efcb80d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbd9e4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efcb80b1300 0x7efcb80b1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45838/0 (socket says 172.21.15.1:45838) 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbc9e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efcb80b2ed0 0x7efcb80d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35356/0 (socket says 172.21.15.1:35356) 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbd9e4640 1 -- 172.21.15.1:0/1331184349 learned_addr learned my addr 172.21.15.1:0/1331184349 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.964+0000 7efcbd1e3640 1 --2- 172.21.15.1:0/1331184349 >> v2:172.21.15.3:3300/0 conn(0x7efcb815e390 0x7efcb80d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.965+0000 7efcbc9e2640 1 -- 172.21.15.1:0/1331184349 >> v2:172.21.15.3:3300/0 conn(0x7efcb815e390 msgr2=0x7efcb80d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.965+0000 7efcbc9e2640 1 --2- 172.21.15.1:0/1331184349 >> v2:172.21.15.3:3300/0 conn(0x7efcb815e390 0x7efcb80d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.965+0000 7efcbc9e2640 1 -- 172.21.15.1:0/1331184349 >> v1:172.21.15.1:6789/0 conn(0x7efcb80b1300 legacy=0x7efcb80b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.965+0000 7efcbc9e2640 1 -- 172.21.15.1:0/1331184349 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb815e770 con 0x7efcb80b2ed0 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.965+0000 7efcbc9e2640 1 --2- 172.21.15.1:0/1331184349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80d2850 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efcb808c020 tx=0x7efca802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa5accb3b51db60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcaffff640 1 -- 172.21.15.1:0/1331184349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8031830 con 0x7efcb80b2ed0 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcaffff640 1 -- 172.21.15.1:0/1331184349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca8031990 con 0x7efcb80b2ed0 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcaffff640 1 -- 172.21.15.1:0/1331184349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8031d20 con 0x7efcb80b2ed0 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 -- 172.21.15.1:0/1331184349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 msgr2=0x7efcb80d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 --2- 172.21.15.1:0/1331184349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80d2850 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efcb808c020 tx=0x7efca802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 -- 172.21.15.1:0/1331184349 shutdown_connections 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 --2- 172.21.15.1:0/1331184349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80d2850 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 --2- 172.21.15.1:0/1331184349 >> v2:172.21.15.3:3300/0 conn(0x7efcb815e390 0x7efcb80d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 -- 172.21.15.1:0/1331184349 >> 172.21.15.1:0/1331184349 conn(0x7efcb80d3ee0 msgr2=0x7efcb80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 -- 172.21.15.1:0/1331184349 shutdown_connections 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.966+0000 7efcbf447640 1 -- 172.21.15.1:0/1331184349 wait complete. 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.967+0000 7efcbf447640 1 Processor -- start 2024-03-24T10:35:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 -- start start 2024-03-24T10:35:07.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efcb80b1300 0x7efcb815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb815e390 0x7efcb80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efcb80b80c0 con 0x7efcb80b2ed0 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efcb80b7780 con 0x7efcb815e390 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbf447640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efcb80c9d20 con 0x7efcb80b1300 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbc9e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbc9e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35358/0 (socket says 172.21.15.1:35358) 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.968+0000 7efcbc9e2640 1 -- 172.21.15.1:0/845031868 learned_addr learned my addr 172.21.15.1:0/845031868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbd9e4640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb815e390 0x7efcb80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbd1e3640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efcb80b1300 0x7efcb815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 -- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efcb80b1300 msgr2=0x7efcb815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efcb80b1300 0x7efcb815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 -- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb815e390 msgr2=0x7efcb80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb815e390 0x7efcb80c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 -- 172.21.15.1:0/845031868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca8009400 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbc9e2640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efca8009890 tx=0x7efca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca803a6f0 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca803a870 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb80b7a00 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca803ac20 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.969+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb80b8a20 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.970+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb80b8200 con 0x7efcb80b2ed0 2024-03-24T10:35:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.971+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efca8043410 con 0x7efcb80b2ed0 2024-03-24T10:35:07.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.971+0000 7efcadffb640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 0x7efc9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:07.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.971+0000 7efcbd1e3640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 0x7efc9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:07.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.971+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7efca80ac0c0 con 0x7efcb80b2ed0 2024-03-24T10:35:07.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.972+0000 7efcbd1e3640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 0x7efc9c06dcd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcb815ed00 tx=0x7efcb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:07.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:07.975+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcb80b8200 con 0x7efcb80b2ed0 2024-03-24T10:35:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.154+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efcb80b8200 con 0x7efcb80b2ed0 2024-03-24T10:35:08.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.155+0000 7efcadffb640 1 -- 172.21.15.1:0/845031868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7efcb80b8200 con 0x7efcb80b2ed0 2024-03-24T10:35:08.155 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 msgr2=0x7efc9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 0x7efc9c06dcd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcb815ed00 tx=0x7efcb4008040 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 msgr2=0x7efcb80b84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efca8009890 tx=0x7efca80046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 shutdown_connections 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc9c06b810 0x7efc9c06dcd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb815e390 0x7efcb80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcb80b2ed0 0x7efcb80b84e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 --2- 172.21.15.1:0/845031868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efcb80b1300 0x7efcb815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 >> 172.21.15.1:0/845031868 conn(0x7efcb80d3ee0 msgr2=0x7efcb80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 shutdown_connections 2024-03-24T10:35:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.157+0000 7efcbf447640 1 -- 172.21.15.1:0/845031868 wait complete. 2024-03-24T10:35:08.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:08.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.289+0000 7f4010925640 1 Processor -- start 2024-03-24T10:35:08.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.289+0000 7f4010925640 1 -- start start 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4010925640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 0x7f400c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4010925640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f400c162610 0x7f400c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4010925640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400c0c7a40 con 0x7f400c0b4e40 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4010925640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f400c0cfd50 con 0x7f400c162610 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4010925640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f400c0b55b0 con 0x7f400c1615f0 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f400ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f400c162610 0x7f400c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f400ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f400c162610 0x7f400c0d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35374/0 (socket says 172.21.15.1:35374) 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f4009d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f400c0b4e40 0x7f400c161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45844/0 (socket says 172.21.15.1:45844) 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f400ad9d640 1 -- 172.21.15.1:0/1894288054 learned_addr learned my addr 172.21.15.1:0/1894288054 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f400a59c640 1 --2- 172.21.15.1:0/1894288054 >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 0x7f400c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.291+0000 7f400ad9d640 1 -- 172.21.15.1:0/1894288054 >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 msgr2=0x7f400c0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.292+0000 7f400ad9d640 1 --2- 172.21.15.1:0/1894288054 >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 0x7f400c0d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.292+0000 7f400ad9d640 1 -- 172.21.15.1:0/1894288054 >> v1:172.21.15.1:6789/0 conn(0x7f400c0b4e40 legacy=0x7f400c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.292+0000 7f400ad9d640 1 -- 172.21.15.1:0/1894288054 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c0b52d0 con 0x7f400c162610 2024-03-24T10:35:08.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.292+0000 7f400a59c640 1 --2- 172.21.15.1:0/1894288054 >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 0x7f400c0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:08.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f400ad9d640 1 --2- 172.21.15.1:0/1894288054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c162610 0x7f400c0d1c70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f400c0b69f0 tx=0x7f400402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4713a96d9b99f0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f400959a640 1 -- 172.21.15.1:0/1894288054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400402cb20 con 0x7f400c162610 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f400959a640 1 -- 172.21.15.1:0/1894288054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f400402cc80 con 0x7f400c162610 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f400959a640 1 -- 172.21.15.1:0/1894288054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40040343f0 con 0x7f400c162610 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 -- 172.21.15.1:0/1894288054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c162610 msgr2=0x7f400c0d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 --2- 172.21.15.1:0/1894288054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c162610 0x7f400c0d1c70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f400c0b69f0 tx=0x7f400402bf80 comp rx=0 tx=0).stop 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 -- 172.21.15.1:0/1894288054 shutdown_connections 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 --2- 172.21.15.1:0/1894288054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c162610 0x7f400c0d1c70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 --2- 172.21.15.1:0/1894288054 >> v2:172.21.15.3:3300/0 conn(0x7f400c1615f0 0x7f400c0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 -- 172.21.15.1:0/1894288054 >> 172.21.15.1:0/1894288054 conn(0x7f400c0d3ef0 msgr2=0x7f400c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 -- 172.21.15.1:0/1894288054 shutdown_connections 2024-03-24T10:35:08.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.293+0000 7f4010925640 1 -- 172.21.15.1:0/1894288054 wait complete. 2024-03-24T10:35:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.295+0000 7f4010925640 1 Processor -- start 2024-03-24T10:35:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.295+0000 7f4010925640 1 -- start start 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f400c162610 0x7f400c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f400c0c8c20 0x7f400c0c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0c95d0 con 0x7f400c1615f0 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0cc300 con 0x7f400c162610 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4010925640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f400c0cbee0 con 0x7f400c0c8c20 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f400a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f400a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35384/0 (socket says 172.21.15.1:35384) 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f400a59c640 1 -- 172.21.15.1:0/3997888399 learned_addr learned my addr 172.21.15.1:0/3997888399 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f4009d9b640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f400c162610 0x7f400c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.296+0000 7f400ad9d640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f400c0c8c20 0x7f400c0c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 -- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f400c0c8c20 msgr2=0x7f400c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f400c0c8c20 0x7f400c0c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 -- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f400c162610 msgr2=0x7f400c156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f400c162610 0x7f400c156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 -- 172.21.15.1:0/3997888399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4004009b30 con 0x7f400c1615f0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f400a59c640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00e7e0 tx=0x7f3ffc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc004030 con 0x7f400c1615f0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ffc004190 con 0x7f400c1615f0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400c0cc5a0 con 0x7f400c1615f0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc004520 con 0x7f400c1615f0 2024-03-24T10:35:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.297+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400c0ccb60 con 0x7f400c1615f0 2024-03-24T10:35:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.298+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f400c04f470 con 0x7f400c1615f0 2024-03-24T10:35:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.298+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ffc00c6d0 con 0x7f400c1615f0 2024-03-24T10:35:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.299+0000 7f3ffaffd640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 0x7f3fec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.299+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f3ffc089fa0 con 0x7f400c1615f0 2024-03-24T10:35:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.299+0000 7f4009d9b640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 0x7f3fec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.299+0000 7f4009d9b640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 0x7f3fec06dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f400c15cf40 tx=0x7f4000007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.303+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ffc0548c0 con 0x7f400c1615f0 2024-03-24T10:35:08.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.485+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f400c08d640 con 0x7f400c1615f0 2024-03-24T10:35:08.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.485+0000 7f3ffaffd640 1 -- 172.21.15.1:0/3997888399 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f3ffc0200a0 con 0x7f400c1615f0 2024-03-24T10:35:08.485 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:08.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 msgr2=0x7f3fec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 0x7f3fec06dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f400c15cf40 tx=0x7f4000007cc0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 msgr2=0x7f400c0cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00e7e0 tx=0x7f3ffc00eca0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 shutdown_connections 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3fec06b810 0x7f3fec06dcd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f400c0c8c20 0x7f400c0c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f400c162610 0x7f400c156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 --2- 172.21.15.1:0/3997888399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f400c1615f0 0x7f400c0cb9a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 >> 172.21.15.1:0/3997888399 conn(0x7f400c0d3ef0 msgr2=0x7f400c06c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 shutdown_connections 2024-03-24T10:35:08.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.487+0000 7f4010925640 1 -- 172.21.15.1:0/3997888399 wait complete. 2024-03-24T10:35:08.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.618+0000 7effdd9d7640 1 Processor -- start 2024-03-24T10:35:08.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.619+0000 7effdd9d7640 1 -- start start 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effdd9d7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effd815c060 0x7effd80d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effdd9d7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 0x7effd80d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effdd9d7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd80b19b0 con 0x7effd80b1e40 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effdd9d7640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7effd80d0070 con 0x7effd815c060 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effdd9d7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7effd80cfdb0 con 0x7effd80d27e0 2024-03-24T10:35:08.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effd6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effd815c060 0x7effd80d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effcffff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 0x7effd80d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effd6ffd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7effd815c060 0x7effd80d02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:52380/0 (socket says 172.21.15.1:52380) 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effd77fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7effd80b1e40 0x7effd80b22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:44418/0 (socket says 172.21.15.1:44418) 2024-03-24T10:35:08.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.620+0000 7effd6ffd640 1 -- 172.21.15.1:0/2495950294 learned_addr learned my addr 172.21.15.1:0/2495950294 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd6ffd640 1 -- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 msgr2=0x7effd80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd6ffd640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 0x7effd80d07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd6ffd640 1 -- 172.21.15.1:0/2495950294 >> v1:172.21.15.3:6789/0 conn(0x7effd80b1e40 legacy=0x7effd80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd6ffd640 1 -- 172.21.15.1:0/2495950294 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd80b1360 con 0x7effd815c060 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd67fc640 1 -- 172.21.15.1:0/2495950294 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162597041 0 0) 0x7effd80b19b0 con 0x7effd80b1e40 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effcffff640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 0x7effd80d07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.621+0000 7effd6ffd640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd815c060 0x7effd80d02a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7effc8009870 tx=0x7effc802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3b55e43a1b32a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effd67fc640 1 -- 172.21.15.1:0/2495950294 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc80317c0 con 0x7effd815c060 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effd67fc640 1 -- 172.21.15.1:0/2495950294 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc8031920 con 0x7effd815c060 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effd67fc640 1 -- 172.21.15.1:0/2495950294 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8031cb0 con 0x7effd815c060 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 -- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd815c060 msgr2=0x7effd80d02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd815c060 0x7effd80d02a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7effc8009870 tx=0x7effc802b860 comp rx=0 tx=0).stop 2024-03-24T10:35:08.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 -- 172.21.15.1:0/2495950294 shutdown_connections 2024-03-24T10:35:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd80d27e0 0x7effd80d07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 --2- 172.21.15.1:0/2495950294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd815c060 0x7effd80d02a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 -- 172.21.15.1:0/2495950294 >> 172.21.15.1:0/2495950294 conn(0x7effd80d1cc0 msgr2=0x7effd80d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 -- 172.21.15.1:0/2495950294 shutdown_connections 2024-03-24T10:35:08.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.622+0000 7effdd9d7640 1 -- 172.21.15.1:0/2495950294 wait complete. 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.623+0000 7effdd9d7640 1 Processor -- start 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.623+0000 7effdd9d7640 1 -- start start 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 0x7effd81f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 0x7effd80afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7effd81f2ab0 con 0x7effd80d27e0 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7effd8156b70 con 0x7effd80b1e40 2024-03-24T10:35:08.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effdd9d7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7effd8156e50 con 0x7effd815c060 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effcffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effcffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35396/0 (socket says 172.21.15.1:35396) 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effcffff640 1 -- 172.21.15.1:0/1174729339 learned_addr learned my addr 172.21.15.1:0/1174729339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effd6ffd640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 0x7effd81f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.624+0000 7effd77fe640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 0x7effd80afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 -- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 msgr2=0x7effd80afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 0x7effd80afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 -- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 msgr2=0x7effd81f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 0x7effd81f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 -- 172.21.15.1:0/1174729339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc0009400 con 0x7effd80d27e0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effd77fe640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 0x7effd80afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effd6ffd640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 0x7effd81f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effcffff640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7effc000e960 tx=0x7effc000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000ca00 con 0x7effd80d27e0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc000cb60 con 0x7effd80d27e0 2024-03-24T10:35:08.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc80094d0 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0013470 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.625+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd8157410 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.626+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd8157630 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.627+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7effc00135d0 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.627+0000 7effd4ff9640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 0x7effb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.627+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7effc008a5a0 con 0x7effd80d27e0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.627+0000 7effd6ffd640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 0x7effb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:08.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.627+0000 7effd6ffd640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 0x7effb806dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effc8036000 tx=0x7effc8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:08.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.631+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7effc0054e40 con 0x7effd80d27e0 2024-03-24T10:35:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.826+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7effd815e7e0 con 0x7effd80d27e0 2024-03-24T10:35:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.826+0000 7effd4ff9640 1 -- 172.21.15.1:0/1174729339 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7effc0058500 con 0x7effd80d27e0 2024-03-24T10:35:08.827 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 msgr2=0x7effb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 0x7effb806dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effc8036000 tx=0x7effc8008030 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 msgr2=0x7effd81562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7effc000e960 tx=0x7effc000ee20 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 shutdown_connections 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7effb806b810 0x7effb806dcd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7effd815c060 0x7effd80afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effd80d27e0 0x7effd81562e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 --2- 172.21.15.1:0/1174729339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd80b1e40 0x7effd81f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 >> 172.21.15.1:0/1174729339 conn(0x7effd80d1cc0 msgr2=0x7effd80ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 shutdown_connections 2024-03-24T10:35:08.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:08.828+0000 7effdd9d7640 1 -- 172.21.15.1:0/1174729339 wait complete. 2024-03-24T10:35:08.839 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:35:08.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:08.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:08.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[52115]: fuse finished with error 0 and tester_r 0 2024-03-24T10:35:08.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:08.945 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:35:08.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:08.945 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:35:08.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:35:09.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.103+0000 7fdc1dc2c640 1 Processor -- start 2024-03-24T10:35:09.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.103+0000 7fdc1dc2c640 1 -- start start 2024-03-24T10:35:09.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc1dc2c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc18154fe0 0x7fdc1806b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc1dc2c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc1806b5e0 0x7fdc180adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc1dc2c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc181553c0 con 0x7fdc180ae1b0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc1dc2c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1806d250 con 0x7fdc18154fe0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc1dc2c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fdc180ae620 con 0x7fdc1806b5e0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc18154fe0 0x7fdc1806b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc18154fe0 0x7fdc1806b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35404/0 (socket says 172.21.15.1:35404) 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 -- 172.21.15.1:0/2412695062 learned_addr learned my addr 172.21.15.1:0/2412695062 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc16ffd640 1 --2- 172.21.15.1:0/2412695062 >> v2:172.21.15.3:3300/0 conn(0x7fdc1806b5e0 0x7fdc180adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 -- 172.21.15.1:0/2412695062 >> v2:172.21.15.3:3300/0 conn(0x7fdc1806b5e0 msgr2=0x7fdc180adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 --2- 172.21.15.1:0/2412695062 >> v2:172.21.15.3:3300/0 conn(0x7fdc1806b5e0 0x7fdc180adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 -- 172.21.15.1:0/2412695062 >> v1:172.21.15.3:6789/0 conn(0x7fdc180ae1b0 legacy=0x7fdc18161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.104+0000 7fdc177fe640 1 -- 172.21.15.1:0/2412695062 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1806c7e0 con 0x7fdc18154fe0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.105+0000 7fdc177fe640 1 --2- 172.21.15.1:0/2412695062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1806b0a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc180b3a20 tx=0x7fdc0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ad93901eb1b064 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.105+0000 7fdc167fc640 1 -- 172.21.15.1:0/2412695062 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08031870 con 0x7fdc18154fe0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.105+0000 7fdc167fc640 1 -- 172.21.15.1:0/2412695062 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc080319d0 con 0x7fdc18154fe0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.105+0000 7fdc167fc640 1 -- 172.21.15.1:0/2412695062 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08031d60 con 0x7fdc18154fe0 2024-03-24T10:35:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2412695062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 msgr2=0x7fdc1806b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2412695062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1806b0a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc180b3a20 tx=0x7fdc0802b910 comp rx=0 tx=0).stop 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2412695062 shutdown_connections 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2412695062 >> v2:172.21.15.3:3300/0 conn(0x7fdc1806b5e0 0x7fdc180adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2412695062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1806b0a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2412695062 >> 172.21.15.1:0/2412695062 conn(0x7fdc180d0570 msgr2=0x7fdc180d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2412695062 shutdown_connections 2024-03-24T10:35:09.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.106+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2412695062 wait complete. 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.107+0000 7fdc1dc2c640 1 Processor -- start 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.107+0000 7fdc1dc2c640 1 -- start start 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc1806b5e0 0x7fdc180cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180ae1b0 0x7fdc181514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc18151060 con 0x7fdc18154fe0 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fdc180cf110 con 0x7fdc180ae1b0 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc1dc2c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdc18140970 con 0x7fdc1806b5e0 2024-03-24T10:35:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35408/0 (socket says 172.21.15.1:35408) 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 -- 172.21.15.1:0/2940371399 learned_addr learned my addr 172.21.15.1:0/2940371399 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc177fe640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc1806b5e0 0x7fdc180cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc16ffd640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180ae1b0 0x7fdc181514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 -- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc1806b5e0 msgr2=0x7fdc180cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc1806b5e0 0x7fdc180cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 -- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180ae1b0 msgr2=0x7fdc181514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180ae1b0 0x7fdc181514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 -- 172.21.15.1:0/2940371399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc080094d0 con 0x7fdc18154fe0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.108+0000 7fdc17fff640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c780 tx=0x7fdc1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.109+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10004b60 con 0x7fdc18154fe0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.109+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1000ce90 con 0x7fdc18154fe0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.109+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc18150730 con 0x7fdc18154fe0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.109+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000a850 con 0x7fdc18154fe0 2024-03-24T10:35:09.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.109+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc18150cf0 con 0x7fdc18154fe0 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.110+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc18151210 con 0x7fdc18154fe0 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.110+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdc1000a9b0 con 0x7fdc18154fe0 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.110+0000 7fdc0ffff640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 0x7fdbf806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.111+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fdc1008a3d0 con 0x7fdc18154fe0 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.111+0000 7fdc177fe640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 0x7fdbf806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:09.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.111+0000 7fdc177fe640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 0x7fdbf806dcd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdc08036000 tx=0x7fdc0803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:09.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.115+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdc18151210 con 0x7fdc18154fe0 2024-03-24T10:35:09.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.484+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdc1804f4e0 con 0x7fdc18154fe0 2024-03-24T10:35:09.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.484+0000 7fdc0ffff640 1 -- 172.21.15.1:0/2940371399 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+82 (secure 0 0 0) 0x7fdc10054d10 con 0x7fdc18154fe0 2024-03-24T10:35:09.487 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-24T10:35:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 msgr2=0x7fdbf806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 0x7fdbf806dcd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdc08036000 tx=0x7fdc0803c040 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 msgr2=0x7fdc1814e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000c780 tx=0x7fdc1000cc40 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 shutdown_connections 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdbf806b810 0x7fdbf806dcd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc18154fe0 0x7fdc1814e740 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180ae1b0 0x7fdc181514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 --2- 172.21.15.1:0/2940371399 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc1806b5e0 0x7fdc180cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 >> 172.21.15.1:0/2940371399 conn(0x7fdc180d0570 msgr2=0x7fdc180b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 shutdown_connections 2024-03-24T10:35:09.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09.486+0000 7fdc1dc2c640 1 -- 172.21.15.1:0/2940371399 wait complete. 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:35:09.498 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:09.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:09.499 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:35:09.514 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:35:09.514 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:35:09.514 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:35:09.514 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.514 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:09.515 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42631sec preferred_lft 42631sec 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:35:09.516 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:35:09.517 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:09.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:09.518 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:35:09.583 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:35:09.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:09.583 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:35:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:35:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:35:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:35:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:35:09.638 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:35:09.638 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:35:09.638 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:35:09.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:35:09.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:35:09.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:35:09.639 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:35:09.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:35:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:09.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:09.775 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:35:09.795 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:35:09.795 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:35:09.795 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:09.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:09.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:35:09.862 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:35:09.862 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:09.862 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:09.916 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:09.917 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:09.917 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:09.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:10.040 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:10.112 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:10.112 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:10.113 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:10.114 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:10.115 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:10.115 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:10.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.115 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:10.167 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:35:10.167 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:35:10.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:10.236 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:10.292 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:10.293 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:10.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.295 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:10.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:35:10.295+0000 7f8c14037480 -1 init, newargv = 0x5611c30bba70 newargc=15 2024-03-24T10:35:10.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[53350]: starting ceph client 2024-03-24T10:35:10.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[53350]: starting fuse 2024-03-24T10:35:10.318 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:35:10.318 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:35:10.319 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:35:10.441 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.53350.asok 2024-03-24T10:35:10.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:10.452 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53350.asok 2024-03-24T10:35:10.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.452 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53350.asok status 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:35:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "53350", 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5604, 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5604 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:10.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2490819441 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2490819441 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.5604 192.168.0.1:0/2490819441", 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2490819441", 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 17, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 37, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:35:10.597 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:35:10.598 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:35:10.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.604 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:35:10.604 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:10.628 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:35:10.628 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:10.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:10.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.698 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-24T10:35:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:10.754 INFO:teuthology.orchestra.run.smithi001.stderr:setfattr: layoutfile: Permission denied 2024-03-24T10:35:10.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:10.755 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-24T10:35:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:10.811 INFO:teuthology.orchestra.run.smithi001.stderr:setfattr: layoutdir: Permission denied 2024-03-24T10:35:10.811 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:10.812 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:10.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.011+0000 7f697cfa7640 1 Processor -- start 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 -- start start 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f69780d2bb0 0x7f69780b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f69780b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697815e770 con 0x7f697815e390 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f69780b3190 con 0x7f69780d2bb0 2024-03-24T10:35:11.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697cfa7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f69780d4e40 con 0x7f69780d1000 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f697815e390 0x7f69780d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58298/0 (socket says 172.21.15.1:58298) 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.012+0000 7f697659c640 1 -- 172.21.15.1:0/3957722785 learned_addr learned my addr 172.21.15.1:0/3957722785 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 --2- 172.21.15.1:0/3957722785 >> v2:172.21.15.3:3300/0 conn(0x7f69780d2bb0 0x7f69780b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6976d9d640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f69780b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128137485 0 0) 0x7f6968001110 con 0x7f697815e390 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6964003520 con 0x7f697815e390 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2975342896 0 0) 0x7f6964003520 con 0x7f697815e390 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 msgr2=0x7f69780b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f69780b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 >> v2:172.21.15.3:3300/0 conn(0x7f69780d2bb0 msgr2=0x7f69780b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 --2- 172.21.15.1:0/3957722785 >> v2:172.21.15.3:3300/0 conn(0x7f69780d2bb0 0x7f69780b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69780d1e40 con 0x7f697815e390 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f6968001800 con 0x7f697815e390 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 >> v1:172.21.15.1:6789/0 conn(0x7f697815e390 legacy=0x7f69780d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f6964003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6964003ef0 0x7f69640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6964004d00 0x7f6964005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f69780d4e40 con 0x7f6964003ef0 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f696400b9e0 con 0x7f6964004d00 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f696400bba0 con 0x7f69780d1000 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6964003ef0 0x7f69640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f697659c640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f6964003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f6976d9d640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6964004d00 0x7f6964005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 -- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 msgr2=0x7f6964003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f6964003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 -- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6964004d00 msgr2=0x7f6964005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.013+0000 7f696ffff640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6964004d00 0x7f6964005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f696ffff640 1 -- 172.21.15.1:0/3957722785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6964003520 con 0x7f6964003ef0 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f697659c640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f69780d1000 0x7f6964003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f6976d9d640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6964004d00 0x7f6964005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f696ffff640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6964003ef0 0x7f69640047c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6960009ac0 tx=0x7f69640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597f7624184df4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696002c710 con 0x7f6964003ef0 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696002c870 con 0x7f6964003ef0 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.014+0000 7f6975d9b640 1 -- 172.21.15.1:0/3957722785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696002cc00 con 0x7f6964003ef0 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 -- 172.21.15.1:0/3957722785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6964003ef0 msgr2=0x7f69640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 --2- 172.21.15.1:0/3957722785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6964003ef0 0x7f69640047c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6960009ac0 tx=0x7f69640057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f6976d9d640 1 -- 172.21.15.1:0/3957722785 reap_dead start 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 -- 172.21.15.1:0/3957722785 shutdown_connections 2024-03-24T10:35:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 -- 172.21.15.1:0/3957722785 >> 172.21.15.1:0/3957722785 conn(0x7f69780d3ec0 msgr2=0x7f69780d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 -- 172.21.15.1:0/3957722785 shutdown_connections 2024-03-24T10:35:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.015+0000 7f697cfa7640 1 -- 172.21.15.1:0/3957722785 wait complete. 2024-03-24T10:35:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.016+0000 7f697cfa7640 1 Processor -- start 2024-03-24T10:35:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.016+0000 7f697cfa7640 1 -- start start 2024-03-24T10:35:11.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6964003ef0 0x7f697814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 0x7f69780b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f69780cb5a0 con 0x7f69780b79c0 2024-03-24T10:35:11.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f69780caf90 con 0x7f6978158000 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697cfa7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f69780b7ed0 con 0x7f6964003ef0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f6976d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f6976d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35760/0 (socket says 172.21.15.1:35760) 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f6976d9d640 1 -- 172.21.15.1:0/460505780 learned_addr learned my addr 172.21.15.1:0/460505780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f696ffff640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 0x7f69780b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.017+0000 7f697659c640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6964003ef0 0x7f697814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 -- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6964003ef0 msgr2=0x7f697814d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6964003ef0 0x7f697814d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 -- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 msgr2=0x7f69780b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 0x7f69780b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 -- 172.21.15.1:0/460505780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69780b81b0 con 0x7f69780b79c0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f696ffff640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 0x7f69780b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f6976d9d640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f697808d2d0 tx=0x7f69700115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697000e030 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970007e60 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69780cb230 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970016470 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.018+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69780cd8a0 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.019+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697804f4e0 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.019+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f69700165f0 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.020+0000 7f696effd640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 0x7f695806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.020+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f697000f070 con 0x7f69780b79c0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.020+0000 7f697659c640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 0x7f695806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.020+0000 7f697659c640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 0x7f695806dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f69780ccd60 tx=0x7f69680011e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.024+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6970055400 con 0x7f69780b79c0 2024-03-24T10:35:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.206+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69780b8340 con 0x7f69780b79c0 2024-03-24T10:35:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.208+0000 7f696effd640 1 -- 172.21.15.1:0/460505780 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f697001a090 con 0x7f69780b79c0 2024-03-24T10:35:11.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:11.208 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 msgr2=0x7f695806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 0x7f695806dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f69780ccd60 tx=0x7f69680011e0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 msgr2=0x7f69780b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f697808d2d0 tx=0x7f69700115a0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 shutdown_connections 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f695806b840 0x7f695806dd00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69780b79c0 0x7f69780b8970 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6978158000 0x7f69780b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 --2- 172.21.15.1:0/460505780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6964003ef0 0x7f697814d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 >> 172.21.15.1:0/460505780 conn(0x7f69780d3ec0 msgr2=0x7f69780b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 shutdown_connections 2024-03-24T10:35:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.210+0000 7f697cfa7640 1 -- 172.21.15.1:0/460505780 wait complete. 2024-03-24T10:35:11.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:11.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.327+0000 7f8054240640 1 Processor -- start 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.327+0000 7f8054240640 1 -- start start 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 0x7f804c0d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f804c0b2ed0 0x7f804c0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 0x7f804c0b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f804c0b0b70 con 0x7f804c0b2ed0 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f804c0d1d60 con 0x7f804c15e390 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8054240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f804c0b0de0 con 0x7f804c0b1300 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f804c0b2ed0 0x7f804c0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f804c0b2ed0 0x7f804c0d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35776/0 (socket says 172.21.15.1:35776) 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 -- 172.21.15.1:0/3742571601 learned_addr learned my addr 172.21.15.1:0/3742571601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f8051fdc640 1 --2- 172.21.15.1:0/3742571601 >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 0x7f804c0d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80527dd640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 0x7f804c0b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 -- 172.21.15.1:0/3742571601 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 msgr2=0x7f804c0b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 0x7f804c0b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 -- 172.21.15.1:0/3742571601 >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 msgr2=0x7f804c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.328+0000 7f80517db640 1 --2- 172.21.15.1:0/3742571601 >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 0x7f804c0d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f80517db640 1 -- 172.21.15.1:0/3742571601 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c15e770 con 0x7f804c0b2ed0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f80527dd640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 0x7f804c0b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f8051fdc640 1 --2- 172.21.15.1:0/3742571601 >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 0x7f804c0d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f80517db640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b2ed0 0x7f804c0d2850 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f804c08c020 tx=0x7f804002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80710208f4eb8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f8050fda640 1 -- 172.21.15.1:0/3742571601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040031860 con 0x7f804c0b2ed0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.329+0000 7f8050fda640 1 -- 172.21.15.1:0/3742571601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80400319c0 con 0x7f804c0b2ed0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8050fda640 1 -- 172.21.15.1:0/3742571601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040031d50 con 0x7f804c0b2ed0 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 -- 172.21.15.1:0/3742571601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b2ed0 msgr2=0x7f804c0d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b2ed0 0x7f804c0d2850 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f804c08c020 tx=0x7f804002b900 comp rx=0 tx=0).stop 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 -- 172.21.15.1:0/3742571601 shutdown_connections 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b1300 0x7f804c0b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 --2- 172.21.15.1:0/3742571601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b2ed0 0x7f804c0d2850 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 --2- 172.21.15.1:0/3742571601 >> v2:172.21.15.3:3300/0 conn(0x7f804c15e390 0x7f804c0d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 -- 172.21.15.1:0/3742571601 >> 172.21.15.1:0/3742571601 conn(0x7f804c0d3ee0 msgr2=0x7f804c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 -- 172.21.15.1:0/3742571601 shutdown_connections 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.330+0000 7f8054240640 1 -- 172.21.15.1:0/3742571601 wait complete. 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.331+0000 7f8054240640 1 Processor -- start 2024-03-24T10:35:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.331+0000 7f8054240640 1 -- start start 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.332+0000 7f8054240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b2ed0 0x7f804c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f804c15e390 0x7f804c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f804c1f4ce0 con 0x7f804c0b1300 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f804c161830 con 0x7f804c15e390 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f804c158b20 con 0x7f804c0b2ed0 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35784/0 (socket says 172.21.15.1:35784) 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 -- 172.21.15.1:0/1933542088 learned_addr learned my addr 172.21.15.1:0/1933542088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f80527dd640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f804c15e390 0x7f804c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f80517db640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b2ed0 0x7f804c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 -- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b2ed0 msgr2=0x7f804c1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b2ed0 0x7f804c1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 -- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f804c15e390 msgr2=0x7f804c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f804c15e390 0x7f804c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 -- 172.21.15.1:0/1933542088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80400094d0 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8051fdc640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f804800d7d0 tx=0x7f804800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048004030 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8048004190 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048004520 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804c160530 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.333+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804c160af0 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.334+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f804c04f4e0 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.335+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f80480096a0 con 0x7f804c0b1300 2024-03-24T10:35:11.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.335+0000 7f803e7fc640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 0x7f803006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.336+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f804808a320 con 0x7f804c0b1300 2024-03-24T10:35:11.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.336+0000 7f80517db640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 0x7f803006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.336+0000 7f80517db640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 0x7f803006dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8040036000 tx=0x7f804003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.340+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8048054cf0 con 0x7f804c0b1300 2024-03-24T10:35:11.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.522+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f804c0d3210 con 0x7f804c0b1300 2024-03-24T10:35:11.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.522+0000 7f803e7fc640 1 -- 172.21.15.1:0/1933542088 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f80480583b0 con 0x7f804c0b1300 2024-03-24T10:35:11.524 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 msgr2=0x7f803006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 0x7f803006dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8040036000 tx=0x7f804003c040 comp rx=0 tx=0).stop 2024-03-24T10:35:11.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 msgr2=0x7f804c1612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f804800d7d0 tx=0x7f804800dc90 comp rx=0 tx=0).stop 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 shutdown_connections 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f803006b810 0x7f803006dcd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f804c15e390 0x7f804c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f804c0b2ed0 0x7f804c1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 --2- 172.21.15.1:0/1933542088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f804c0b1300 0x7f804c1612f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 >> 172.21.15.1:0/1933542088 conn(0x7f804c0d3ee0 msgr2=0x7f804c0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 shutdown_connections 2024-03-24T10:35:11.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.524+0000 7f8054240640 1 -- 172.21.15.1:0/1933542088 wait complete. 2024-03-24T10:35:11.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:11.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.640+0000 7f948b64e640 1 Processor -- start 2024-03-24T10:35:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f948b64e640 1 -- start start 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f948b64e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94840af3f0 0x7f9484162600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f948b64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948415eb50 con 0x7f948415e600 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f948b64e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948415ee30 con 0x7f94840b4fe0 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f948b64e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9484162b40 con 0x7f94840af3f0 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.641+0000 7f94893ea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94840af3f0 0x7f9484162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f9488be9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f948415e600 0x7f94840af7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58314/0 (socket says 172.21.15.1:58314) 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f9488be9640 1 -- 172.21.15.1:0/1501479171 learned_addr learned my addr 172.21.15.1:0/1501479171 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f94893ea640 1 -- 172.21.15.1:0/1501479171 >> v1:172.21.15.3:6789/0 conn(0x7f94840b4fe0 legacy=0x7f94840b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f94893ea640 1 -- 172.21.15.1:0/1501479171 >> v1:172.21.15.1:6789/0 conn(0x7f948415e600 legacy=0x7f94840af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f94893ea640 1 -- 172.21.15.1:0/1501479171 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948415f110 con 0x7f94840af3f0 2024-03-24T10:35:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.642+0000 7f947bfff640 1 -- 172.21.15.1:0/1501479171 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111530348 0 0) 0x7f948415eb50 con 0x7f948415e600 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f94893ea640 1 --2- 172.21.15.1:0/1501479171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94840af3f0 0x7f9484162600 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f94800097a0 tx=0x7f948002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8918662520a4da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f947bfff640 1 -- 172.21.15.1:0/1501479171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948002e070 con 0x7f94840af3f0 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f947bfff640 1 -- 172.21.15.1:0/1501479171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94800315a0 con 0x7f94840af3f0 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f947bfff640 1 -- 172.21.15.1:0/1501479171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480031930 con 0x7f94840af3f0 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f948b64e640 1 -- 172.21.15.1:0/1501479171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94840af3f0 msgr2=0x7f9484162600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f948b64e640 1 --2- 172.21.15.1:0/1501479171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94840af3f0 0x7f9484162600 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f94800097a0 tx=0x7f948002b840 comp rx=0 tx=0).stop 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f948b64e640 1 -- 172.21.15.1:0/1501479171 shutdown_connections 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f948b64e640 1 --2- 172.21.15.1:0/1501479171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94840af3f0 0x7f9484162600 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.643+0000 7f948b64e640 1 -- 172.21.15.1:0/1501479171 >> 172.21.15.1:0/1501479171 conn(0x7f9484072410 msgr2=0x7f948406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.644+0000 7f948b64e640 1 -- 172.21.15.1:0/1501479171 shutdown_connections 2024-03-24T10:35:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.644+0000 7f948b64e640 1 -- 172.21.15.1:0/1501479171 wait complete. 2024-03-24T10:35:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 Processor -- start 2024-03-24T10:35:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 -- start start 2024-03-24T10:35:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94840af3f0 0x7f94840cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94840b4fe0 0x7f94841599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9484152a20 con 0x7f948415e600 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9484152740 con 0x7f94840b4fe0 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f948b64e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9484152d80 con 0x7f94840af3f0 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f9489beb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f9489beb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35812/0 (socket says 172.21.15.1:35812) 2024-03-24T10:35:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.645+0000 7f9489beb640 1 -- 172.21.15.1:0/2612485099 learned_addr learned my addr 172.21.15.1:0/2612485099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9488be9640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94840b4fe0 0x7f94841599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 -- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94840af3f0 msgr2=0x7f94840cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94840af3f0 0x7f94840cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 -- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94840b4fe0 msgr2=0x7f94841599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94840b4fe0 0x7f94841599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 -- 172.21.15.1:0/2612485099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9480009400 con 0x7f948415e600 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9489beb640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f947c00c780 tx=0x7f947c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c004b60 con 0x7f948415e600 2024-03-24T10:35:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f947c00ce90 con 0x7f948415e600 2024-03-24T10:35:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9484153060 con 0x7f948415e600 2024-03-24T10:35:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c00a850 con 0x7f948415e600 2024-03-24T10:35:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.646+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948414f010 con 0x7f948415e600 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.647+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9484153580 con 0x7f948415e600 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.648+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f947c00a9b0 con 0x7f948415e600 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.648+0000 7f9479ffb640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 0x7f946806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.648+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f947c08a510 con 0x7f948415e600 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.649+0000 7f94893ea640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 0x7f946806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.649+0000 7f94893ea640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 0x7f946806dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9480036000 tx=0x7f9480030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.656+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9484153580 con 0x7f948415e600 2024-03-24T10:35:11.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.837+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9484153580 con 0x7f948415e600 2024-03-24T10:35:11.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.837+0000 7f9479ffb640 1 -- 172.21.15.1:0/2612485099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f9484153580 con 0x7f948415e600 2024-03-24T10:35:11.838 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 msgr2=0x7f946806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 0x7f946806dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9480036000 tx=0x7f9480030000 comp rx=0 tx=0).stop 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 msgr2=0x7f948414ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f947c00c780 tx=0x7f947c00cc40 comp rx=0 tx=0).stop 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 shutdown_connections 2024-03-24T10:35:11.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f946806b810 0x7f946806dcd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f948415e600 0x7f948414ead0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94840b4fe0 0x7f94841599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 --2- 172.21.15.1:0/2612485099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f94840af3f0 0x7f94840cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 >> 172.21.15.1:0/2612485099 conn(0x7f9484072410 msgr2=0x7f948415c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 shutdown_connections 2024-03-24T10:35:11.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.839+0000 7f948b64e640 1 -- 172.21.15.1:0/2612485099 wait complete. 2024-03-24T10:35:11.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:11.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 Processor -- start 2024-03-24T10:35:11.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 -- start start 2024-03-24T10:35:11.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f47840b2ed0 0x7f47840d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f47840b1300 0x7f47840b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.955+0000 7f4789ec0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f47840b0b70 con 0x7f47840b1300 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4789ec0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f47840d1d60 con 0x7f47840b2ed0 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4789ec0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f47840b0de0 con 0x7f478415e390 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f47840b1300 0x7f47840b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f47840b1300 0x7f47840b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35828/0 (socket says 172.21.15.1:35828) 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 -- 172.21.15.1:0/539463275 learned_addr learned my addr 172.21.15.1:0/539463275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4782ffd640 1 --2- 172.21.15.1:0/539463275 >> v2:172.21.15.3:3300/0 conn(0x7f47840b2ed0 0x7f47840d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f47837fe640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 -- 172.21.15.1:0/539463275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 msgr2=0x7f47840d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 -- 172.21.15.1:0/539463275 >> v2:172.21.15.3:3300/0 conn(0x7f47840b2ed0 msgr2=0x7f47840d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 --2- 172.21.15.1:0/539463275 >> v2:172.21.15.3:3300/0 conn(0x7f47840b2ed0 0x7f47840d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.956+0000 7f4783fff640 1 -- 172.21.15.1:0/539463275 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478415e770 con 0x7f47840b1300 2024-03-24T10:35:11.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.957+0000 7f4783fff640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47840b1790 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f478408c810 tx=0x7f477c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86869761c299ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.957+0000 7f47827fc640 1 -- 172.21.15.1:0/539463275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c02ca70 con 0x7f47840b1300 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.957+0000 7f47827fc640 1 -- 172.21.15.1:0/539463275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f477c02cbd0 con 0x7f47840b1300 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.957+0000 7f47827fc640 1 -- 172.21.15.1:0/539463275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c0343f0 con 0x7f47840b1300 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 -- 172.21.15.1:0/539463275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 msgr2=0x7f47840b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47840b1790 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f478408c810 tx=0x7f477c02bed0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 -- 172.21.15.1:0/539463275 shutdown_connections 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47840b1790 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 --2- 172.21.15.1:0/539463275 >> v2:172.21.15.3:3300/0 conn(0x7f47840b2ed0 0x7f47840d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 --2- 172.21.15.1:0/539463275 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 -- 172.21.15.1:0/539463275 >> 172.21.15.1:0/539463275 conn(0x7f47840d3ee0 msgr2=0x7f47840d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 -- 172.21.15.1:0/539463275 shutdown_connections 2024-03-24T10:35:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.958+0000 7f4789ec0640 1 -- 172.21.15.1:0/539463275 wait complete. 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.959+0000 7f4789ec0640 1 Processor -- start 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.959+0000 7f4789ec0640 1 -- start start 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47840b2ed0 0x7f47840b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f47840b8a70 con 0x7f47840b1300 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f478409f3f0 con 0x7f47840b2ed0 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4789ec0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f47840b7ef0 con 0x7f478415e390 2024-03-24T10:35:11.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35838/0 (socket says 172.21.15.1:35838) 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 -- 172.21.15.1:0/3525173561 learned_addr learned my addr 172.21.15.1:0/3525173561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4782ffd640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47840b2ed0 0x7f47840b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f4783fff640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 -- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 msgr2=0x7f47840cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 -- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47840b2ed0 msgr2=0x7f47840b7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47840b2ed0 0x7f47840b7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.960+0000 7f47837fe640 1 -- 172.21.15.1:0/3525173561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c009b30 con 0x7f47840b1300 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f47837fe640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f477400e870 tx=0x7f477400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740094f0 con 0x7f47840b1300 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f477400c940 con 0x7f47840b1300 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47840b81d0 con 0x7f47840b1300 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477400ccf0 con 0x7f47840b1300 2024-03-24T10:35:11.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.961+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47840b8790 con 0x7f47840b1300 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.962+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f478404f4e0 con 0x7f47840b1300 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.962+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4774013470 con 0x7f47840b1300 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.963+0000 7f476bfff640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 0x7f477006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.963+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f477408a460 con 0x7f47840b1300 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.963+0000 7f4782ffd640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 0x7f477006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:11.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.963+0000 7f4782ffd640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 0x7f477006dd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f47840d01f0 tx=0x7f4778008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:11.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:11.967+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4774054c70 con 0x7f47840b1300 2024-03-24T10:35:12.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.148+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f478409f560 con 0x7f47840b1300 2024-03-24T10:35:12.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.148+0000 7f476bfff640 1 -- 172.21.15.1:0/3525173561 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7f478409f560 con 0x7f47840b1300 2024-03-24T10:35:12.149 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 msgr2=0x7f477006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 0x7f477006dd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f47840d01f0 tx=0x7f4778008040 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 msgr2=0x7f47841542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f477400e870 tx=0x7f477400ed30 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 shutdown_connections 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477006b840 0x7f477006dd00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f478415e390 0x7f47840cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47840b2ed0 0x7f47840b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.150+0000 7f4789ec0640 1 --2- 172.21.15.1:0/3525173561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47840b1300 0x7f47841542f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.151+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 >> 172.21.15.1:0/3525173561 conn(0x7f47840d3ee0 msgr2=0x7f478406cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.151+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 shutdown_connections 2024-03-24T10:35:12.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.151+0000 7f4789ec0640 1 -- 172.21.15.1:0/3525173561 wait complete. 2024-03-24T10:35:12.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 Processor -- start 2024-03-24T10:35:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 -- start start 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 0x7f85a00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85a00b4e40 0x7f85a0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a00c7a40 con 0x7f85a01619d0 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f85a00cfd50 con 0x7f85a00b4e40 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f85a7064640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f85a00b55b0 con 0x7f85a01615f0 2024-03-24T10:35:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.267+0000 7f8597fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85a00b4e40 0x7f85a0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85a00b4e40 0x7f85a0161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35850/0 (socket says 172.21.15.1:35850) 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 -- 172.21.15.1:0/1677320474 learned_addr learned my addr 172.21.15.1:0/1677320474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f85a4e00640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 0x7f85a00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 -- 172.21.15.1:0/1677320474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 msgr2=0x7f85a00d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 0x7f85a00d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 -- 172.21.15.1:0/1677320474 >> v1:172.21.15.1:6789/0 conn(0x7f85a01619d0 legacy=0x7f85a00d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f8597fff640 1 -- 172.21.15.1:0/1677320474 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a00b52d0 con 0x7f85a00b4e40 2024-03-24T10:35:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.268+0000 7f85a4e00640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 0x7f85a00d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:12.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.269+0000 7f8597fff640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a00b4e40 0x7f85a0161f00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8590009870 tx=0x7f859002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b7a2b5ed5a9b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.269+0000 7f85977fe640 1 -- 172.21.15.1:0/1677320474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590030840 con 0x7f85a00b4e40 2024-03-24T10:35:12.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.269+0000 7f85977fe640 1 -- 172.21.15.1:0/1677320474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85900309a0 con 0x7f85a00b4e40 2024-03-24T10:35:12.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.269+0000 7f85977fe640 1 -- 172.21.15.1:0/1677320474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590030d30 con 0x7f85a00b4e40 2024-03-24T10:35:12.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 -- 172.21.15.1:0/1677320474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a00b4e40 msgr2=0x7f85a0161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a00b4e40 0x7f85a0161f00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8590009870 tx=0x7f859002b980 comp rx=0 tx=0).stop 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 -- 172.21.15.1:0/1677320474 shutdown_connections 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a00b4e40 0x7f85a0161f00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 --2- 172.21.15.1:0/1677320474 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a01615f0 0x7f85a00d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 -- 172.21.15.1:0/1677320474 >> 172.21.15.1:0/1677320474 conn(0x7f85a00d3ef0 msgr2=0x7f85a00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 -- 172.21.15.1:0/1677320474 shutdown_connections 2024-03-24T10:35:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.270+0000 7f85a7064640 1 -- 172.21.15.1:0/1677320474 wait complete. 2024-03-24T10:35:12.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.272+0000 7f85a7064640 1 Processor -- start 2024-03-24T10:35:12.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.272+0000 7f85a7064640 1 -- start start 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 0x7f85a0160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 0x7f85a00ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f85a01f3f70 con 0x7f85a01615f0 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f85a01f4230 con 0x7f85a01619d0 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a7064640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f85a01f4510 con 0x7f85a0159930 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a4e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a4e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35864/0 (socket says 172.21.15.1:35864) 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.273+0000 7f85a4e00640 1 -- 172.21.15.1:0/3441603128 learned_addr learned my addr 172.21.15.1:0/3441603128 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f8597fff640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 0x7f85a0160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a5601640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 0x7f85a00ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 -- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 msgr2=0x7f85a00ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 0x7f85a00ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 -- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 msgr2=0x7f85a0160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 0x7f85a0160710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 -- 172.21.15.1:0/3441603128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85900094d0 con 0x7f85a01615f0 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f8597fff640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 0x7f85a0160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a5601640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 0x7f85a00ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a4e00640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f859c009fc0 tx=0x7f859c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c003b90 con 0x7f85a01615f0 2024-03-24T10:35:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f859c00f040 con 0x7f85a01615f0 2024-03-24T10:35:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01f47f0 con 0x7f85a01615f0 2024-03-24T10:35:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c004030 con 0x7f85a01615f0 2024-03-24T10:35:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.274+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a00b7970 con 0x7f85a01615f0 2024-03-24T10:35:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.275+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a00b7b00 con 0x7f85a01615f0 2024-03-24T10:35:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.275+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f859c004190 con 0x7f85a01615f0 2024-03-24T10:35:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.276+0000 7f85957fa640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 0x7f858c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.276+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f859c08a610 con 0x7f85a01615f0 2024-03-24T10:35:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.276+0000 7f8597fff640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 0x7f858c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.276+0000 7f8597fff640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 0x7f858c06dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8590036000 tx=0x7f859003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.280+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f859c054f50 con 0x7f85a01615f0 2024-03-24T10:35:12.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.460+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f85a004f470 con 0x7f85a01615f0 2024-03-24T10:35:12.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.461+0000 7f85957fa640 1 -- 172.21.15.1:0/3441603128 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f859c058610 con 0x7f85a01615f0 2024-03-24T10:35:12.461 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 msgr2=0x7f858c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 0x7f858c06dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8590036000 tx=0x7f859003b040 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 msgr2=0x7f85a01601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f859c009fc0 tx=0x7f859c00d560 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 shutdown_connections 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f858c06b810 0x7f858c06dcd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f85a0159930 0x7f85a00ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85a01619d0 0x7f85a0160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 --2- 172.21.15.1:0/3441603128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a01615f0 0x7f85a01601d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 >> 172.21.15.1:0/3441603128 conn(0x7f85a00d3ef0 msgr2=0x7f85a006c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 shutdown_connections 2024-03-24T10:35:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.463+0000 7f85a7064640 1 -- 172.21.15.1:0/3441603128 wait complete. 2024-03-24T10:35:12.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:12.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.577+0000 7fcc0ebdd640 1 Processor -- start 2024-03-24T10:35:12.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.577+0000 7fcc0ebdd640 1 -- start start 2024-03-24T10:35:12.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0ebdd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcc080d2bb0 0x7fcc080b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0ebdd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc080d1000 0x7fcc080b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0ebdd640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0815e770 con 0x7fcc0815e390 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0ebdd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcc080b3190 con 0x7fcc080d1000 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0ebdd640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fcc080d4e40 con 0x7fcc080d2bb0 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0d17a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc080d1000 0x7fcc080b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0d17a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc080d1000 0x7fcc080b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35866/0 (socket says 172.21.15.1:35866) 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.578+0000 7fcc0d17a640 1 -- 172.21.15.1:0/198729623 learned_addr learned my addr 172.21.15.1:0/198729623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc07fff640 1 --2- 172.21.15.1:0/198729623 >> v2:172.21.15.3:3300/0 conn(0x7fcc080d2bb0 0x7fcc080b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc0d17a640 1 -- 172.21.15.1:0/198729623 >> v2:172.21.15.3:3300/0 conn(0x7fcc080d2bb0 msgr2=0x7fcc080b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc0d17a640 1 --2- 172.21.15.1:0/198729623 >> v2:172.21.15.3:3300/0 conn(0x7fcc080d2bb0 0x7fcc080b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc0d17a640 1 -- 172.21.15.1:0/198729623 >> v1:172.21.15.3:6789/0 conn(0x7fcc0815e390 legacy=0x7fcc080d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc0d17a640 1 -- 172.21.15.1:0/198729623 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc080d1e40 con 0x7fcc080d1000 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.579+0000 7fcc0d17a640 1 --2- 172.21.15.1:0/198729623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc080d1000 0x7fcc080b40d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcc0806b4e0 tx=0x7fcc0002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc29ad96f7ed205 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc077fe640 1 -- 172.21.15.1:0/198729623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0002c960 con 0x7fcc080d1000 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc077fe640 1 -- 172.21.15.1:0/198729623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0002cac0 con 0x7fcc080d1000 2024-03-24T10:35:12.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc077fe640 1 -- 172.21.15.1:0/198729623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0002ce50 con 0x7fcc080d1000 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/198729623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc080d1000 msgr2=0x7fcc080b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/198729623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc080d1000 0x7fcc080b40d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcc0806b4e0 tx=0x7fcc0002bdc0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/198729623 shutdown_connections 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/198729623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc080d1000 0x7fcc080b40d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/198729623 >> v2:172.21.15.3:3300/0 conn(0x7fcc080d2bb0 0x7fcc080b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/198729623 >> 172.21.15.1:0/198729623 conn(0x7fcc080d3ec0 msgr2=0x7fcc080d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/198729623 shutdown_connections 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.580+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/198729623 wait complete. 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.582+0000 7fcc0ebdd640 1 Processor -- start 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.582+0000 7fcc0ebdd640 1 -- start start 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc080d1000 0x7fcc080b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 0x7fcc080b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcc080b9490 con 0x7fcc0815e390 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fcc080b8880 con 0x7fcc080d1000 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0ebdd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcc080b8b60 con 0x7fcc080d2bb0 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0d17a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0d17a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35882/0 (socket says 172.21.15.1:35882) 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0d17a640 1 -- 172.21.15.1:0/2230346190 learned_addr learned my addr 172.21.15.1:0/2230346190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc0c979640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc080d1000 0x7fcc080b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.583+0000 7fcc07fff640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 0x7fcc080b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 -- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 msgr2=0x7fcc080b7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 0x7fcc080b7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 -- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc080d1000 msgr2=0x7fcc080b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc080d1000 0x7fcc080b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 -- 172.21.15.1:0/2230346190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc00009b30 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc07fff640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 0x7fcc080b7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0d17a640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcc00036040 tx=0x7fcc00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0003e450 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0003e5d0 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0003e980 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc080b8e40 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.584+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc080b9920 con 0x7fcc0815e390 2024-03-24T10:35:12.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.585+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0804f4e0 con 0x7fcc0815e390 2024-03-24T10:35:12.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.586+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc0003eae0 con 0x7fcc0815e390 2024-03-24T10:35:12.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.586+0000 7fcc057fa640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 0x7fcbe806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.586+0000 7fcc0c979640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 0x7fcbe806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.586+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fcc000acfa0 con 0x7fcc0815e390 2024-03-24T10:35:12.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.587+0000 7fcc0c979640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 0x7fcbe806dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcc080a1de0 tx=0x7fcbf8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.591+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc000e0e10 con 0x7fcc0815e390 2024-03-24T10:35:12.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.783+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcc0815e770 con 0x7fcc0815e390 2024-03-24T10:35:12.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.783+0000 7fcc057fa640 1 -- 172.21.15.1:0/2230346190 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7fcc0815e770 con 0x7fcc0815e390 2024-03-24T10:35:12.783 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 msgr2=0x7fcbe806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 0x7fcbe806dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcc080a1de0 tx=0x7fcbf8009060 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 msgr2=0x7fcc081f7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcc00036040 tx=0x7fcc00004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 shutdown_connections 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcbe806b810 0x7fcbe806dcd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0815e390 0x7fcc081f7080 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcc080d2bb0 0x7fcc080b7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 --2- 172.21.15.1:0/2230346190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcc080d1000 0x7fcc080b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 >> 172.21.15.1:0/2230346190 conn(0x7fcc080d3ec0 msgr2=0x7fcc080b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 shutdown_connections 2024-03-24T10:35:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.785+0000 7fcc0ebdd640 1 -- 172.21.15.1:0/2230346190 wait complete. 2024-03-24T10:35:12.796 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:12.796 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:12.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.957+0000 7f64ed015640 1 Processor -- start 2024-03-24T10:35:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.957+0000 7f64ed015640 1 -- start start 2024-03-24T10:35:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.958+0000 7f64ed015640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64e815e390 0x7f64e80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.958+0000 7f64ed015640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.958+0000 7f64ed015640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e815e770 con 0x7f64e80b3320 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.958+0000 7f64ed015640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f64e80b3190 con 0x7f64e815e390 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.958+0000 7f64ed015640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f64e80d4e40 con 0x7f64e80d1000 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64e815e390 0x7f64e80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64e815e390 0x7f64e80d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35886/0 (socket says 172.21.15.1:35886) 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 -- 172.21.15.1:0/746937845 learned_addr learned my addr 172.21.15.1:0/746937845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e659c640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 -- 172.21.15.1:0/746937845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 msgr2=0x7f64e80d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 -- 172.21.15.1:0/746937845 >> v1:172.21.15.3:6789/0 conn(0x7f64e80b3320 legacy=0x7f64e80cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.959+0000 7f64e6d9d640 1 -- 172.21.15.1:0/746937845 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e80d1e40 con 0x7f64e815e390 2024-03-24T10:35:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64e6d9d640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e815e390 0x7f64e80d24a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009870 tx=0x7f64e002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6401de57de2609 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64e5d9b640 1 -- 172.21.15.1:0/746937845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e00317c0 con 0x7f64e815e390 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64e5d9b640 1 -- 172.21.15.1:0/746937845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0031920 con 0x7f64e815e390 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64e5d9b640 1 -- 172.21.15.1:0/746937845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0031cb0 con 0x7f64e815e390 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 -- 172.21.15.1:0/746937845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e815e390 msgr2=0x7f64e80d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e815e390 0x7f64e80d24a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009870 tx=0x7f64e002b860 comp rx=0 tx=0).stop 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 -- 172.21.15.1:0/746937845 shutdown_connections 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 --2- 172.21.15.1:0/746937845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e815e390 0x7f64e80d24a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 -- 172.21.15.1:0/746937845 >> 172.21.15.1:0/746937845 conn(0x7f64e80d3ec0 msgr2=0x7f64e80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 -- 172.21.15.1:0/746937845 shutdown_connections 2024-03-24T10:35:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.960+0000 7f64ed015640 1 -- 172.21.15.1:0/746937845 wait complete. 2024-03-24T10:35:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.963+0000 7f64ed015640 1 Processor -- start 2024-03-24T10:35:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.963+0000 7f64ed015640 1 -- start start 2024-03-24T10:35:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64e815e390 0x7f64e80d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80b90b0 con 0x7f64e80b3320 2024-03-24T10:35:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f64e815bd30 con 0x7f64e815e390 2024-03-24T10:35:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64ed015640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f64e815f3e0 con 0x7f64e80d1000 2024-03-24T10:35:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35896/0 (socket says 172.21.15.1:35896) 2024-03-24T10:35:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 -- 172.21.15.1:0/3643869496 learned_addr learned my addr 172.21.15.1:0/3643869496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e659c640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e759e640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64e815e390 0x7f64e80d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 -- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 msgr2=0x7f64e80b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 -- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64e815e390 msgr2=0x7f64e80d2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64e815e390 0x7f64e80d2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.964+0000 7f64e6d9d640 1 -- 172.21.15.1:0/3643869496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e00094d0 con 0x7f64e80b3320 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64e6d9d640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f64e002b8a0 tx=0x7f64e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0032860 con 0x7f64e80b3320 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0032a00 con 0x7f64e80b3320 2024-03-24T10:35:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0032db0 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e815eab0 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.965+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e815f070 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.966+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e804f4e0 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.966+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f64e004e450 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.966+0000 7f64df7fe640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 0x7f64c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.967+0000 7f64e659c640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 0x7f64c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.967+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f64e00ac390 con 0x7f64e80b3320 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.967+0000 7f64e659c640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 0x7f64c806dd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b87b0 tx=0x7f64d0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:12.971+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f64e0076d70 con 0x7f64e80b3320 2024-03-24T10:35:13.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.153+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64e80ca820 con 0x7f64e80b3320 2024-03-24T10:35:13.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.154+0000 7f64df7fe640 1 -- 172.21.15.1:0/3643869496 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f64e007a430 con 0x7f64e80b3320 2024-03-24T10:35:13.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:13.158 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 msgr2=0x7f64c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 0x7f64c806dd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b87b0 tx=0x7f64d0009190 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 msgr2=0x7f64e815b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f64e002b8a0 tx=0x7f64e00046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 shutdown_connections 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f64c806b840 0x7f64c806dd00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64e815e390 0x7f64e80d2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f64e80d1000 0x7f64e80b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 --2- 172.21.15.1:0/3643869496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64e80b3320 0x7f64e815b7f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 >> 172.21.15.1:0/3643869496 conn(0x7f64e80d3ec0 msgr2=0x7f64e80cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 shutdown_connections 2024-03-24T10:35:13.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.156+0000 7f64ed015640 1 -- 172.21.15.1:0/3643869496 wait complete. 2024-03-24T10:35:13.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:13.170 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.275+0000 7f8a6adf6640 1 Processor -- start 2024-03-24T10:35:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.275+0000 7f8a6adf6640 1 -- start start 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a6adf6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a640d2a20 0x7f8a640cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a6adf6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a640b0b70 con 0x7f8a6415e390 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a6adf6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a640d1d60 con 0x7f8a640b2310 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a6adf6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a640b0de0 con 0x7f8a640d2a20 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a63fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a640d2a20 0x7f8a640cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a68b92640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a6415e390 0x7f8a640d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58334/0 (socket says 172.21.15.1:58334) 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.276+0000 7f8a68b92640 1 -- 172.21.15.1:0/667637136 learned_addr learned my addr 172.21.15.1:0/667637136 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a63fff640 1 -- 172.21.15.1:0/667637136 >> v1:172.21.15.3:6789/0 conn(0x7f8a640b2310 legacy=0x7f8a640cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a63fff640 1 -- 172.21.15.1:0/667637136 >> v1:172.21.15.1:6789/0 conn(0x7f8a6415e390 legacy=0x7f8a640d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a637fe640 1 -- 172.21.15.1:0/667637136 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1261616207 0 0) 0x7f8a640b0b70 con 0x7f8a6415e390 2024-03-24T10:35:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a63fff640 1 -- 172.21.15.1:0/667637136 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6415e770 con 0x7f8a640d2a20 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a63fff640 1 --2- 172.21.15.1:0/667637136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640d2a20 0x7f8a640cead0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a255ba90cb54c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a637fe640 1 -- 172.21.15.1:0/667637136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c030610 con 0x7f8a640d2a20 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a637fe640 1 -- 172.21.15.1:0/667637136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4c030770 con 0x7f8a640d2a20 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.277+0000 7f8a637fe640 1 -- 172.21.15.1:0/667637136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c030b00 con 0x7f8a640d2a20 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 -- 172.21.15.1:0/667637136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640d2a20 msgr2=0x7f8a640cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/667637136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640d2a20 0x7f8a640cead0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b960 comp rx=0 tx=0).stop 2024-03-24T10:35:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 -- 172.21.15.1:0/667637136 shutdown_connections 2024-03-24T10:35:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/667637136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640d2a20 0x7f8a640cead0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 -- 172.21.15.1:0/667637136 >> 172.21.15.1:0/667637136 conn(0x7f8a640d3ee0 msgr2=0x7f8a640d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 -- 172.21.15.1:0/667637136 shutdown_connections 2024-03-24T10:35:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.278+0000 7f8a6adf6640 1 -- 172.21.15.1:0/667637136 wait complete. 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.279+0000 7f8a6adf6640 1 Processor -- start 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.279+0000 7f8a6adf6640 1 -- start start 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 0x7f8a640b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 0x7f8a641f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a641f73d0 con 0x7f8a640b2310 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8a641f76b0 con 0x7f8a6415e390 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a6adf6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8a641f7990 con 0x7f8a640d2a20 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35916/0 (socket says 172.21.15.1:35916) 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 -- 172.21.15.1:0/2164073988 learned_addr learned my addr 172.21.15.1:0/2164073988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a69393640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 0x7f8a641f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a63fff640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 0x7f8a640b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 -- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 msgr2=0x7f8a640b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 0x7f8a640b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 -- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 msgr2=0x7f8a641f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 0x7f8a641f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a68b92640 1 -- 172.21.15.1:0/2164073988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c009400 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a69393640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 0x7f8a641f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.280+0000 7f8a63fff640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 0x7f8a640b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a68b92640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400c3f0 tx=0x7f8a5400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a540074e0 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a54007640 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a641f7c70 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54004a00 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.281+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a641f6ff0 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.282+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6404f4e0 con 0x7f8a640b2310 2024-03-24T10:35:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.282+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8a54004b60 con 0x7f8a640b2310 2024-03-24T10:35:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.283+0000 7f8a617fa640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 0x7f8a4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.283+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f8a5408a160 con 0x7f8a640b2310 2024-03-24T10:35:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.283+0000 7f8a63fff640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 0x7f8a4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.283+0000 7f8a63fff640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 0x7f8a4406dcd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c036000 tx=0x7f8a4c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.287+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8a54054a80 con 0x7f8a640b2310 2024-03-24T10:35:13.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.468+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8a640d3210 con 0x7f8a640b2310 2024-03-24T10:35:13.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.468+0000 7f8a617fa640 1 -- 172.21.15.1:0/2164073988 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f8a54058140 con 0x7f8a640b2310 2024-03-24T10:35:13.469 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:13.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 msgr2=0x7f8a4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 0x7f8a4406dcd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c036000 tx=0x7f8a4c03b040 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 msgr2=0x7f8a640b81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400c3f0 tx=0x7f8a5400c8b0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 shutdown_connections 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a4406b810 0x7f8a4406dcd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6415e390 0x7f8a641f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a640d2a20 0x7f8a640b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 --2- 172.21.15.1:0/2164073988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a640b2310 0x7f8a640b81e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 >> 172.21.15.1:0/2164073988 conn(0x7f8a640d3ee0 msgr2=0x7f8a640d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 shutdown_connections 2024-03-24T10:35:13.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.470+0000 7f8a6adf6640 1 -- 172.21.15.1:0/2164073988 wait complete. 2024-03-24T10:35:13.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.589+0000 7feb9c7e2640 1 Processor -- start 2024-03-24T10:35:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.589+0000 7feb9c7e2640 1 -- start start 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9c7e2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb940cbf90 0x7feb9415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9c7e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb940d0690 0x7feb9406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9c7e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb940b45c0 con 0x7feb9415e860 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9c7e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7feb9406c930 con 0x7feb940d0690 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9c7e2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7feb9415ed00 con 0x7feb940cbf90 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.590+0000 7feb9ad7f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feb940d0690 0x7feb9406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb99d7d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb9415e860 0x7feb940cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58344/0 (socket says 172.21.15.1:58344) 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb99d7d640 1 -- 172.21.15.1:0/823080431 learned_addr learned my addr 172.21.15.1:0/823080431 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9a57e640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb940cbf90 0x7feb9415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9ad7f640 1 -- 172.21.15.1:0/823080431 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb940cbf90 msgr2=0x7feb9415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9ad7f640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb940cbf90 0x7feb9415bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9ad7f640 1 -- 172.21.15.1:0/823080431 >> v1:172.21.15.1:6789/0 conn(0x7feb9415e860 legacy=0x7feb940cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9957c640 1 -- 172.21.15.1:0/823080431 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2793580635 0 0) 0x7feb940b45c0 con 0x7feb9415e860 2024-03-24T10:35:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.591+0000 7feb9ad7f640 1 -- 172.21.15.1:0/823080431 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9415efe0 con 0x7feb940d0690 2024-03-24T10:35:13.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9ad7f640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb9406cd50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009dc0 tx=0x7feb8c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a261ad4eefd1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9957c640 1 -- 172.21.15.1:0/823080431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c02cbb0 con 0x7feb940d0690 2024-03-24T10:35:13.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9957c640 1 -- 172.21.15.1:0/823080431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8c02cd10 con 0x7feb940d0690 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9957c640 1 -- 172.21.15.1:0/823080431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c034660 con 0x7feb940d0690 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 -- 172.21.15.1:0/823080431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 msgr2=0x7feb9406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb9406cd50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009dc0 tx=0x7feb8c02be00 comp rx=0 tx=0).stop 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 -- 172.21.15.1:0/823080431 shutdown_connections 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb9406cd50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/823080431 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb940cbf90 0x7feb9415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 -- 172.21.15.1:0/823080431 >> 172.21.15.1:0/823080431 conn(0x7feb9406b4f0 msgr2=0x7feb9406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 -- 172.21.15.1:0/823080431 shutdown_connections 2024-03-24T10:35:13.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.592+0000 7feb9c7e2640 1 -- 172.21.15.1:0/823080431 wait complete. 2024-03-24T10:35:13.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.593+0000 7feb9c7e2640 1 Processor -- start 2024-03-24T10:35:13.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 -- start start 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb940cbf90 0x7feb940cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb9415e860 0x7feb94159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb940cdec0 con 0x7feb940d0690 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7feb9414d2e0 con 0x7feb940cbf90 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9c7e2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7feb9414d5c0 con 0x7feb9415e860 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb99d7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb99d7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35936/0 (socket says 172.21.15.1:35936) 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb99d7d640 1 -- 172.21.15.1:0/3703442685 learned_addr learned my addr 172.21.15.1:0/3703442685 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9a57e640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb940cbf90 0x7feb940cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.594+0000 7feb9ad7f640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb9415e860 0x7feb94159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 -- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb9415e860 msgr2=0x7feb94159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb9415e860 0x7feb94159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 -- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb940cbf90 msgr2=0x7feb940cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb940cbf90 0x7feb940cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 -- 172.21.15.1:0/3703442685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8c009a70 con 0x7feb940d0690 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb99d7d640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c3f0 tx=0x7feb8800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb880074e0 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb88007640 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9414d8a0 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88004a00 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.595+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9414de60 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.596+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb9414e050 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.597+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feb88004b60 con 0x7feb940d0690 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.597+0000 7feb86ffd640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 0x7feb7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.597+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7feb88089f50 con 0x7feb940d0690 2024-03-24T10:35:13.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.597+0000 7feb9a57e640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 0x7feb7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.598+0000 7feb9a57e640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 0x7feb7806dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feb9000aac0 tx=0x7feb90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.601+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feb88054920 con 0x7feb940d0690 2024-03-24T10:35:13.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.781+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feb9404f470 con 0x7feb940d0690 2024-03-24T10:35:13.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.781+0000 7feb86ffd640 1 -- 172.21.15.1:0/3703442685 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7feb88057fe0 con 0x7feb940d0690 2024-03-24T10:35:13.784 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 msgr2=0x7feb7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 0x7feb7806dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feb9000aac0 tx=0x7feb90008040 comp rx=0 tx=0).stop 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 msgr2=0x7feb94145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c3f0 tx=0x7feb8800c8b0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 shutdown_connections 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7feb7806b810 0x7feb7806dcd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7feb9415e860 0x7feb94159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb940d0690 0x7feb94145330 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 --2- 172.21.15.1:0/3703442685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb940cbf90 0x7feb940cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 >> 172.21.15.1:0/3703442685 conn(0x7feb9406b4f0 msgr2=0x7feb94070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 shutdown_connections 2024-03-24T10:35:13.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.783+0000 7feb9c7e2640 1 -- 172.21.15.1:0/3703442685 wait complete. 2024-03-24T10:35:13.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.901+0000 7fc10bb0f640 1 Processor -- start 2024-03-24T10:35:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.901+0000 7fc10bb0f640 1 -- start start 2024-03-24T10:35:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10bb0f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 0x7fc1040d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10bb0f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1040cfae0 0x7fc1040d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10bb0f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1040b0b70 con 0x7fc1040b2ed0 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10bb0f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc1040d1d60 con 0x7fc1040cfae0 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10bb0f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc1040b0de0 con 0x7fc10415e390 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1040cfae0 0x7fc1040d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1040cfae0 0x7fc1040d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35944/0 (socket says 172.21.15.1:35944) 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 -- 172.21.15.1:0/3142330459 learned_addr learned my addr 172.21.15.1:0/3142330459 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc1098ab640 1 --2- 172.21.15.1:0/3142330459 >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 0x7fc1040d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 -- 172.21.15.1:0/3142330459 >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 msgr2=0x7fc1040d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 --2- 172.21.15.1:0/3142330459 >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 0x7fc1040d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 -- 172.21.15.1:0/3142330459 >> v1:172.21.15.3:6789/0 conn(0x7fc1040b2ed0 legacy=0x7fc1040b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc10a0ac640 1 -- 172.21.15.1:0/3142330459 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10415e770 con 0x7fc1040cfae0 2024-03-24T10:35:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.902+0000 7fc1098ab640 1 --2- 172.21.15.1:0/3142330459 >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 0x7fc1040d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.903+0000 7fc10a0ac640 1 --2- 172.21.15.1:0/3142330459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1040cfae0 0x7fc1040d2850 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc10408c020 tx=0x7fc0fc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74637a850c753913 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.903+0000 7fc1088a9640 1 -- 172.21.15.1:0/3142330459 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc02cb20 con 0x7fc1040cfae0 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.903+0000 7fc1088a9640 1 -- 172.21.15.1:0/3142330459 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc02cc80 con 0x7fc1040cfae0 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.903+0000 7fc1088a9640 1 -- 172.21.15.1:0/3142330459 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0343f0 con 0x7fc1040cfae0 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 -- 172.21.15.1:0/3142330459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1040cfae0 msgr2=0x7fc1040d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/3142330459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1040cfae0 0x7fc1040d2850 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc10408c020 tx=0x7fc0fc02bf80 comp rx=0 tx=0).stop 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 -- 172.21.15.1:0/3142330459 shutdown_connections 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/3142330459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1040cfae0 0x7fc1040d2850 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/3142330459 >> v2:172.21.15.3:3300/0 conn(0x7fc10415e390 0x7fc1040d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 -- 172.21.15.1:0/3142330459 >> 172.21.15.1:0/3142330459 conn(0x7fc1040d3ee0 msgr2=0x7fc1040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 -- 172.21.15.1:0/3142330459 shutdown_connections 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.904+0000 7fc10bb0f640 1 -- 172.21.15.1:0/3142330459 wait complete. 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.905+0000 7fc10bb0f640 1 Processor -- start 2024-03-24T10:35:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.905+0000 7fc10bb0f640 1 -- start start 2024-03-24T10:35:13.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1040b2ed0 0x7fc1040b8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1040cfae0 0x7fc1040b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc10415d830 con 0x7fc10415e390 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc1040cda00 con 0x7fc1040b2ed0 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc1040cdce0 con 0x7fc1040cfae0 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35946/0 (socket says 172.21.15.1:35946) 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 -- 172.21.15.1:0/2872231759 learned_addr learned my addr 172.21.15.1:0/2872231759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc1098ab640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1040b2ed0 0x7fc1040b8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc1090aa640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1040cfae0 0x7fc1040b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 -- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1040cfae0 msgr2=0x7fc1040b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1040cfae0 0x7fc1040b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 -- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1040b2ed0 msgr2=0x7fc1040b8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1040b2ed0 0x7fc1040b8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 -- 172.21.15.1:0/2872231759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc009b30 con 0x7fc10415e390 2024-03-24T10:35:13.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10a0ac640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036040 tx=0x7fc0fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc007e70 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc009510 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc03e690 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1040cdfc0 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.906+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1040b9440 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.907+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc104068b40 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.908+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc0fc03e7f0 con 0x7fc10415e390 2024-03-24T10:35:13.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.908+0000 7fc0fa7fc640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 0x7fc0e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:13.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.909+0000 7fc1098ab640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 0x7fc0e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:13.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.909+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fc0fc0acd90 con 0x7fc10415e390 2024-03-24T10:35:13.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.909+0000 7fc1098ab640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 0x7fc0e806dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc100006fd0 tx=0x7fc100008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:13.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:13.913+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc0fc077760 con 0x7fc10415e390 2024-03-24T10:35:14.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.127+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1040b95d0 con 0x7fc10415e390 2024-03-24T10:35:14.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.128+0000 7fc0fa7fc640 1 -- 172.21.15.1:0/2872231759 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7fc0fc043020 con 0x7fc10415e390 2024-03-24T10:35:14.128 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 msgr2=0x7fc0e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 0x7fc0e806dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc100006fd0 tx=0x7fc100008040 comp rx=0 tx=0).stop 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 msgr2=0x7fc1040b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036040 tx=0x7fc0fc004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 shutdown_connections 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc0e806b810 0x7fc0e806dcd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10415e390 0x7fc1040b6320 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc1040cfae0 0x7fc1040b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 --2- 172.21.15.1:0/2872231759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1040b2ed0 0x7fc1040b8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 >> 172.21.15.1:0/2872231759 conn(0x7fc1040d3ee0 msgr2=0x7fc10406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 shutdown_connections 2024-03-24T10:35:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.130+0000 7fc10bb0f640 1 -- 172.21.15.1:0/2872231759 wait complete. 2024-03-24T10:35:14.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.251+0000 7f5b1c0b5640 1 Processor -- start 2024-03-24T10:35:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.251+0000 7f5b1c0b5640 1 -- start start 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1c0b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b140d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1c0b5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5b14162610 0x7f5b140d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1c0b5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b140c7a40 con 0x7f5b140b4e40 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1c0b5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f5b140cfd50 con 0x7f5b14162610 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1c0b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5b140b55b0 con 0x7f5b141615f0 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b19650640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b140b4e40 0x7f5b14161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58346/0 (socket says 172.21.15.1:58346) 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b19650640 1 -- 172.21.15.1:0/1071188097 learned_addr learned my addr 172.21.15.1:0/1071188097 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b1a652640 1 --2- 172.21.15.1:0/1071188097 >> v2:172.21.15.3:3300/0 conn(0x7f5b14162610 0x7f5b140d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b140d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346640287 0 0) 0x7f5b140c7a40 con 0x7f5b140b4e40 2024-03-24T10:35:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.252+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b00003520 con 0x7f5b140b4e40 2024-03-24T10:35:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2301994210 0 0) 0x7f5b00003520 con 0x7f5b140b4e40 2024-03-24T10:35:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 msgr2=0x7f5b140d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b140d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 >> v2:172.21.15.3:3300/0 conn(0x7f5b14162610 msgr2=0x7f5b140d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 --2- 172.21.15.1:0/1071188097 >> v2:172.21.15.3:3300/0 conn(0x7f5b14162610 0x7f5b140d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b140b52d0 con 0x7f5b140b4e40 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f5b04001670 con 0x7f5b140b4e40 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 >> v1:172.21.15.1:6789/0 conn(0x7f5b140b4e40 legacy=0x7f5b14161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b00003970 0x7f5b00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b00004d00 0x7f5b00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b140cfd50 con 0x7f5b00003970 2024-03-24T10:35:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5b140b55b0 con 0x7f5b00004d00 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5b0000b980 con 0x7f5b141615f0 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b00003970 0x7f5b00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b1a652640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b00004d00 0x7f5b00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19650640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 -- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 msgr2=0x7f5b000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:14.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 -- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b00004d00 msgr2=0x7f5b00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b00004d00 0x7f5b00005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19e51640 1 -- 172.21.15.1:0/1071188097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b00003520 con 0x7f5b00003970 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.253+0000 7f5b19650640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b141615f0 0x7f5b000047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.254+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b00003970 0x7f5b00003d90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009ac0 tx=0x7f5b000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14465368d9032dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.254+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1002c6b0 con 0x7f5b00003970 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.254+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b1002c810 con 0x7f5b00003970 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.254+0000 7f5b18e4f640 1 -- 172.21.15.1:0/1071188097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1002cba0 con 0x7f5b00003970 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1071188097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b00003970 msgr2=0x7f5b00003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1071188097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b00003970 0x7f5b00003d90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009ac0 tx=0x7f5b000057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1a652640 1 -- 172.21.15.1:0/1071188097 reap_dead start 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1071188097 shutdown_connections 2024-03-24T10:35:14.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1071188097 >> 172.21.15.1:0/1071188097 conn(0x7f5b140d3ef0 msgr2=0x7f5b140d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1071188097 shutdown_connections 2024-03-24T10:35:14.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.255+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1071188097 wait complete. 2024-03-24T10:35:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.256+0000 7f5b1c0b5640 1 Processor -- start 2024-03-24T10:35:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.256+0000 7f5b1c0b5640 1 -- start start 2024-03-24T10:35:14.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b00003970 0x7f5b14158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b140b7fc0 0x7f5b140b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14157a40 con 0x7f5b14154ca0 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5b140b8980 con 0x7f5b140b7fc0 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5b14155520 con 0x7f5b00003970 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35970/0 (socket says 172.21.15.1:35970) 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 -- 172.21.15.1:0/1784751488 learned_addr learned my addr 172.21.15.1:0/1784751488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:14.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b00003970 0x7f5b14158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1a652640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b140b7fc0 0x7f5b140b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 -- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b00003970 msgr2=0x7f5b14158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b00003970 0x7f5b14158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 -- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b140b7fc0 msgr2=0x7f5b140b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b140b7fc0 0x7f5b140b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 -- 172.21.15.1:0/1784751488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b140cc4e0 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b19650640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e850 tx=0x7f5b0400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0400c870 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0400c9d0 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.257+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b14159350 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.258+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0400cd40 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.258+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b14157760 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.259+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b1404f470 con 0x7f5b14154ca0 2024-03-24T10:35:14.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.259+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5b0401d460 con 0x7f5b14154ca0 2024-03-24T10:35:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.259+0000 7f5b0a7fc640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 0x7f5af806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.259+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f5b0408a230 con 0x7f5b14154ca0 2024-03-24T10:35:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.259+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 0x7f5af806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.260+0000 7f5b19e51640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 0x7f5af806dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5b10036000 tx=0x7f5b10030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.264+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5b04054b70 con 0x7f5b14154ca0 2024-03-24T10:35:14.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.502+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f5b1408d640 con 0x7f5b14154ca0 2024-03-24T10:35:14.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.503+0000 7f5b0a7fc640 1 -- 172.21.15.1:0/1784751488 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f5b0400cea0 con 0x7f5b14154ca0 2024-03-24T10:35:14.503 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 msgr2=0x7f5af806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 0x7f5af806dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5b10036000 tx=0x7f5b10030000 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 msgr2=0x7f5b140c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e850 tx=0x7f5b0400ed10 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 shutdown_connections 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5af806b810 0x7f5af806dcd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b140b7fc0 0x7f5b140b8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b14154ca0 0x7f5b140c9b70 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 --2- 172.21.15.1:0/1784751488 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5b00003970 0x7f5b14158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 >> 172.21.15.1:0/1784751488 conn(0x7f5b140d3ef0 msgr2=0x7f5b140b9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 shutdown_connections 2024-03-24T10:35:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.505+0000 7f5b1c0b5640 1 -- 172.21.15.1:0/1784751488 wait complete. 2024-03-24T10:35:14.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:14.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.625+0000 7fe6bb9e2640 1 Processor -- start 2024-03-24T10:35:14.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6bb9e2640 1 -- start start 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6bb9e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6bb9e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b415e770 con 0x7fe6b40d1000 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6bb9e2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b40d4e40 con 0x7fe6b415e390 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6bb9e2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b40b3190 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6b8f7d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6b8f7d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35982/0 (socket says 172.21.15.1:35982) 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/2684893645 learned_addr learned my addr 172.21.15.1:0/2684893645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.626+0000 7fe6b9f7f640 1 --1- 172.21.15.1:0/2684893645 >> v1:172.21.15.1:6789/0 conn(0x7fe6b40d1000 0x7fe6b40b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58352/0 (socket says 172.21.15.1:58352) 2024-03-24T10:35:14.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.627+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/2684893645 >> v1:172.21.15.3:6789/0 conn(0x7fe6b415e390 legacy=0x7fe6b40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:14.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.627+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/2684893645 >> v1:172.21.15.1:6789/0 conn(0x7fe6b40d1000 legacy=0x7fe6b40b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.627+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/2684893645 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b40d1e40 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.627+0000 7fe6abfff640 1 -- 172.21.15.1:0/2684893645 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1052837980 0 0) 0x7fe6ac001490 con 0x7fe6b40d1000 2024-03-24T10:35:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6b8f7d640 1 --2- 172.21.15.1:0/2684893645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009870 tx=0x7fe6a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb105a5970ff20b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6abfff640 1 -- 172.21.15.1:0/2684893645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a40317e0 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6abfff640 1 -- 172.21.15.1:0/2684893645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4031940 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6abfff640 1 -- 172.21.15.1:0/2684893645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4031cd0 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/2684893645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 msgr2=0x7fe6b40b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/2684893645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009870 tx=0x7fe6a402b880 comp rx=0 tx=0).stop 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/2684893645 shutdown_connections 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/2684893645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b40b3b90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/2684893645 >> 172.21.15.1:0/2684893645 conn(0x7fe6b40d3ec0 msgr2=0x7fe6b40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/2684893645 shutdown_connections 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.628+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/2684893645 wait complete. 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.629+0000 7fe6bb9e2640 1 Processor -- start 2024-03-24T10:35:14.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 -- start start 2024-03-24T10:35:14.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe6b40d1000 0x7fe6b41f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6b415e390 0x7fe6b4151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b40ccf40 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b40cd1e0 con 0x7fe6b415e390 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6bb9e2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b41520b0 con 0x7fe6b40d1000 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6b8f7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6b8f7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35994/0 (socket says 172.21.15.1:35994) 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.630+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/3855884117 learned_addr learned my addr 172.21.15.1:0/3855884117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b977e640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe6b40d1000 0x7fe6b41f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b9f7f640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6b415e390 0x7fe6b4151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe6b40d1000 msgr2=0x7fe6b41f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe6b40d1000 0x7fe6b41f47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6b415e390 msgr2=0x7fe6b4151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6b415e390 0x7fe6b4151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 -- 172.21.15.1:0/3855884117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a40094d0 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6b8f7d640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4000c00 tx=0x7fe6a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4032900 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4032a80 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b4152380 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4032e30 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.631+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b4152940 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.632+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b4152b30 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.633+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6a404e450 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.633+0000 7fe6a9ffb640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 0x7fe69806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:14.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.634+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fe6a40ac4a0 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.634+0000 7fe6b977e640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 0x7fe69806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:14.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.634+0000 7fe6b977e640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 0x7fe69806dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b9a70 tx=0x7fe6b0009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:14.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.638+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6a4076f90 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.832+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe6b40cd350 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.832+0000 7fe6a9ffb640 1 -- 172.21.15.1:0/3855884117 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7fe6a407a650 con 0x7fe6b40d2bb0 2024-03-24T10:35:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 msgr2=0x7fe69806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 0x7fe69806dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b9a70 tx=0x7fe6b0009060 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 msgr2=0x7fe6b41559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4000c00 tx=0x7fe6a40046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 shutdown_connections 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe69806b810 0x7fe69806dcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6b415e390 0x7fe6b4151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6b40d2bb0 0x7fe6b41559a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 --2- 172.21.15.1:0/3855884117 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe6b40d1000 0x7fe6b41f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 >> 172.21.15.1:0/3855884117 conn(0x7fe6b40d3ec0 msgr2=0x7fe6b40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 shutdown_connections 2024-03-24T10:35:14.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:14.834+0000 7fe6bb9e2640 1 -- 172.21.15.1:0/3855884117 wait complete. 2024-03-24T10:35:14.846 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:35:14.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:14.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:14.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[53350]: fuse finished with error 0 and tester_r 0 2024-03-24T10:35:14.881 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:20.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:35:20.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:20.984 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:35:21.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-24T10:35:21.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.149+0000 7fe87354f640 1 Processor -- start 2024-03-24T10:35:21.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.149+0000 7fe87354f640 1 -- start start 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe87354f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe86c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe87354f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86c0c7a40 con 0x7fe86c1615f0 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe87354f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86c0b55b0 con 0x7fe86c0b4e40 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe87354f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0cfd50 con 0x7fe86c162610 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe8712eb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe86c1615f0 0x7fe86c161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33332/0 (socket says 172.21.15.1:33332) 2024-03-24T10:35:21.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe8712eb640 1 -- 172.21.15.1:0/213505730 learned_addr learned my addr 172.21.15.1:0/213505730 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe871aec640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe86c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267708938 0 0) 0x7fe86c0c7a40 con 0x7fe86c1615f0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe858003520 con 0x7fe86c1615f0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3532646270 0 0) 0x7fe858003520 con 0x7fe86c1615f0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 msgr2=0x7fe86c0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe86c0d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 >> v1:172.21.15.3:6789/0 conn(0x7fe86c0b4e40 legacy=0x7fe86c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c0b52d0 con 0x7fe86c1615f0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.150+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fe868001670 con 0x7fe86c1615f0 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 >> v1:172.21.15.1:6789/0 conn(0x7fe86c1615f0 legacy=0x7fe86c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8580039a0 0x7fe858003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe8580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe858004d00 0x7fe858005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0cfd50 con 0x7fe8580039a0 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe85800b980 con 0x7fe858004d00 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe85800bb40 con 0x7fe86c162610 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8580039a0 0x7fe858003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe871aec640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe858004d00 0x7fe858005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe870aea640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe8580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 -- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 msgr2=0x7fe8580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe86c162610 0x7fe8580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 -- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe858004d00 msgr2=0x7fe858005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe858004d00 0x7fe858005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.151+0000 7fe8712eb640 1 -- 172.21.15.1:0/213505730 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe858003520 con 0x7fe8580039a0 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.152+0000 7fe8712eb640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8580039a0 0x7fe858003d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c190 tx=0x7fe8580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8073570e2710f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.152+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868005ad0 con 0x7fe8580039a0 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.152+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe868005c50 con 0x7fe8580039a0 2024-03-24T10:35:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.152+0000 7fe863fff640 1 -- 172.21.15.1:0/213505730 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86802c790 con 0x7fe8580039a0 2024-03-24T10:35:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 -- 172.21.15.1:0/213505730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8580039a0 msgr2=0x7fe858003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 --2- 172.21.15.1:0/213505730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8580039a0 0x7fe858003d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c190 tx=0x7fe8580057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe871aec640 1 -- 172.21.15.1:0/213505730 reap_dead start 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 -- 172.21.15.1:0/213505730 shutdown_connections 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 -- 172.21.15.1:0/213505730 >> 172.21.15.1:0/213505730 conn(0x7fe86c0d3ef0 msgr2=0x7fe86c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 -- 172.21.15.1:0/213505730 shutdown_connections 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.153+0000 7fe87354f640 1 -- 172.21.15.1:0/213505730 wait complete. 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.154+0000 7fe87354f640 1 Processor -- start 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.154+0000 7fe87354f640 1 -- start start 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8580039a0 0x7fe86c15a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe86c1f8150 0x7fe86c1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0b88b0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c15aab0 con 0x7fe86c1f8150 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe87354f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c15ad90 con 0x7fe8580039a0 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49152/0 (socket says 172.21.15.1:49152) 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 -- 172.21.15.1:0/1885993223 learned_addr learned my addr 172.21.15.1:0/1885993223 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe8712eb640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8580039a0 0x7fe86c15a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe870aea640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe86c1f8150 0x7fe86c1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 -- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8580039a0 msgr2=0x7fe86c15a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8580039a0 0x7fe86c15a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 -- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe86c1f8150 msgr2=0x7fe86c1552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe86c1f8150 0x7fe86c1552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 -- 172.21.15.1:0/1885993223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c0b81f0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe871aec640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe86c073780 tx=0x7fe86400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.155+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86400e040 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.156+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86400f040 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.156+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c0b84d0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.156+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864003bf0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.156+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c155830 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.157+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c0b89f0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.157+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe864003d50 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.158+0000 7fe861ffb640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.158+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fe86408a9f0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.158+0000 7fe8712eb640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.158+0000 7fe8712eb640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 0x7fe85006dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe868034000 tx=0x7fe868009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.162+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe86c0b89f0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.412+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7fe86c0b8660 con 0x7fe86c1f3f20 2024-03-24T10:35:21.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.414+0000 7fe861ffb640 1 -- 172.21.15.1:0/1885993223 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v15) ==== 211+0+168 (secure 0 0 0) 0x7fe8640553c0 con 0x7fe86c1f3f20 2024-03-24T10:35:21.416 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:35:21.417 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:35:21.417 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow rwp" 2024-03-24T10:35:21.417 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:35:21.417 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-24T10:35:21.417 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 msgr2=0x7fe85006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 0x7fe85006dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe868034000 tx=0x7fe868009030 comp rx=0 tx=0).stop 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 msgr2=0x7fe86c0c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe86c073780 tx=0x7fe86400f580 comp rx=0 tx=0).stop 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 shutdown_connections 2024-03-24T10:35:21.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86c1f3f20 0x7fe86c0c8b60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe86c1f8150 0x7fe86c1552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 --2- 172.21.15.1:0/1885993223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8580039a0 0x7fe86c15a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 >> 172.21.15.1:0/1885993223 conn(0x7fe86c0d3ef0 msgr2=0x7fe86c160920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 shutdown_connections 2024-03-24T10:35:21.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.417+0000 7fe87354f640 1 -- 172.21.15.1:0/1885993223 wait complete. 2024-03-24T10:35:21.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:35:21.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.530+0000 7efc7cc4e640 1 Processor -- start 2024-03-24T10:35:21.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.530+0000 7efc7cc4e640 1 -- start start 2024-03-24T10:35:21.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc7cc4e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efc78161610 0x7efc78162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc7cc4e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc78162b90 con 0x7efc7806f7f0 2024-03-24T10:35:21.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc7cc4e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc780b45a0 con 0x7efc7806ee20 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc7cc4e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7efc781619f0 con 0x7efc78161610 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc76d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efc7806f7f0 0x7efc780cc2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33334/0 (socket says 172.21.15.1:33334) 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc76d9d640 1 -- 172.21.15.1:0/2768172936 learned_addr learned my addr 172.21.15.1:0/2768172936 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.531+0000 7efc7659c640 1 --2- 172.21.15.1:0/2768172936 >> v2:172.21.15.3:3300/0 conn(0x7efc78161610 0x7efc78162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.532+0000 7efc7559a640 1 -- 172.21.15.1:0/2768172936 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686411184 0 0) 0x7efc68001110 con 0x7efc7806f7f0 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.532+0000 7efc7559a640 1 -- 172.21.15.1:0/2768172936 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc60003520 con 0x7efc7806f7f0 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.532+0000 7efc7659c640 1 -- 172.21.15.1:0/2768172936 >> v1:172.21.15.3:6789/0 conn(0x7efc7806ee20 legacy=0x7efc780d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.532+0000 7efc7659c640 1 -- 172.21.15.1:0/2768172936 >> v1:172.21.15.1:6789/0 conn(0x7efc7806f7f0 legacy=0x7efc780cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.532+0000 7efc7659c640 1 -- 172.21.15.1:0/2768172936 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7806f2c0 con 0x7efc78161610 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.533+0000 7efc7659c640 1 --2- 172.21.15.1:0/2768172936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc78161610 0x7efc78162650 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efc700097a0 tx=0x7efc7002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12063fe8e883087e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.533+0000 7efc7559a640 1 -- 172.21.15.1:0/2768172936 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70030610 con 0x7efc78161610 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.533+0000 7efc7559a640 1 -- 172.21.15.1:0/2768172936 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc70030770 con 0x7efc78161610 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.533+0000 7efc7559a640 1 -- 172.21.15.1:0/2768172936 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70030ae0 con 0x7efc78161610 2024-03-24T10:35:21.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 -- 172.21.15.1:0/2768172936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc78161610 msgr2=0x7efc78162650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/2768172936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc78161610 0x7efc78162650 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efc700097a0 tx=0x7efc7002b960 comp rx=0 tx=0).stop 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 -- 172.21.15.1:0/2768172936 shutdown_connections 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/2768172936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc78161610 0x7efc78162650 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 -- 172.21.15.1:0/2768172936 >> 172.21.15.1:0/2768172936 conn(0x7efc780b3770 msgr2=0x7efc780b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 -- 172.21.15.1:0/2768172936 shutdown_connections 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.534+0000 7efc7cc4e640 1 -- 172.21.15.1:0/2768172936 wait complete. 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.535+0000 7efc7cc4e640 1 Processor -- start 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.535+0000 7efc7cc4e640 1 -- start start 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc7806ee20 0x7efc780cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc7806f7f0 0x7efc780b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc781f4d30 con 0x7efc78161610 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efc780b4d30 con 0x7efc7806f7f0 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7cc4e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efc780b5010 con 0x7efc7806ee20 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49154/0 (socket says 172.21.15.1:49154) 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 -- 172.21.15.1:0/3573883435 learned_addr learned my addr 172.21.15.1:0/3573883435 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc7659c640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc7806ee20 0x7efc780cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc75d9b640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc7806f7f0 0x7efc780b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 -- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc7806ee20 msgr2=0x7efc780cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc7806ee20 0x7efc780cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 -- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc7806f7f0 msgr2=0x7efc780b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc7806f7f0 0x7efc780b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.536+0000 7efc76d9d640 1 -- 172.21.15.1:0/3573883435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc70009400 con 0x7efc78161610 2024-03-24T10:35:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc76d9d640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c730 tx=0x7efc6800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68005ab0 con 0x7efc78161610 2024-03-24T10:35:21.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc68005c10 con 0x7efc78161610 2024-03-24T10:35:21.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc780caf30 con 0x7efc78161610 2024-03-24T10:35:21.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68001d50 con 0x7efc78161610 2024-03-24T10:35:21.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.537+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc780cb4f0 con 0x7efc78161610 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.538+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc780b51c0 con 0x7efc78161610 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.539+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efc68007070 con 0x7efc78161610 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.539+0000 7efc66ffd640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 0x7efc5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.539+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7efc68089650 con 0x7efc78161610 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.539+0000 7efc7659c640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 0x7efc5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.540+0000 7efc7659c640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 0x7efc5806dd00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc70036000 tx=0x7efc70008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.543+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc780b51c0 con 0x7efc78161610 2024-03-24T10:35:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.762+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efc78162b90 con 0x7efc78161610 2024-03-24T10:35:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.763+0000 7efc66ffd640 1 -- 172.21.15.1:0/3573883435 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+82 (secure 0 0 0) 0x7efc78162b90 con 0x7efc78161610 2024-03-24T10:35:21.764 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 msgr2=0x7efc5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 0x7efc5806dd00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc70036000 tx=0x7efc70008030 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 msgr2=0x7efc780cded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c730 tx=0x7efc6800cbf0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 shutdown_connections 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efc5806b840 0x7efc5806dd00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc78161610 0x7efc780cded0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc7806f7f0 0x7efc780b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 --2- 172.21.15.1:0/3573883435 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efc7806ee20 0x7efc780cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 >> 172.21.15.1:0/3573883435 conn(0x7efc780b3770 msgr2=0x7efc7815c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 shutdown_connections 2024-03-24T10:35:21.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:21.765+0000 7efc7cc4e640 1 -- 172.21.15.1:0/3573883435 wait complete. 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:35:21.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:35:21.779 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:35:21.779 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:21.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:21.779 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:35:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:35:21.794 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:35:21.794 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42619sec preferred_lft 42619sec 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:35:21.795 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:35:21.797 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:35:21.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:21.797 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:35:21.863 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:35:21.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:21.865 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:35:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:35:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:35:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:35:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:35:21.922 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:35:21.923 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:35:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:35:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:22.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.057 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:35:22.072 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:35:22.073 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:35:22.073 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:22.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:35:22.140 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:35:22.140 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:22.140 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:22.194 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:35:22.194 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:35:22.195 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:22.249 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:22.317 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:22.373 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:22.374 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:22.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.375 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:22.428 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:35:22.428 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:35:22.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:35:22.500 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:35:22.557 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:35:22.558 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:35:22.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.559 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:35:22.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:35:22.557+0000 7f0903a3b480 -1 init, newargv = 0x55ab955f4780 newargc=15 2024-03-24T10:35:22.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[54414]: starting ceph client 2024-03-24T10:35:22.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[54414]: starting fuse 2024-03-24T10:35:22.578 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:35:22.578 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:35:22.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:35:22.579 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:35:22.580 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:35:22.699 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.54414.asok 2024-03-24T10:35:22.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:35:22.710 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54414.asok 2024-03-24T10:35:22.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54414.asok status 2024-03-24T10:35:22.848 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:35:22.848 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:35:22.848 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "54414", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5709, 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5709 2024-03-24T10:35:22.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1966133053 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1966133053 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.5709 192.168.0.1:0/1966133053", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1966133053", 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:35:22.850 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 17, 2024-03-24T10:35:22.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 37, 2024-03-24T10:35:22.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:35:22.851 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:35:22.851 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:35:22.851 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:35:22.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.857 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:35:22.857 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:22.873 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:35:22.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:22.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.873 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:22.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.943 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-24T10:35:22.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:22.999 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-24T10:35:23.059 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:23.059 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:23.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.258+0000 7faa04f25640 1 Processor -- start 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.258+0000 7faa04f25640 1 -- start start 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.259+0000 7faa04f25640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa0015e390 0x7faa000d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.259+0000 7faa04f25640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7faa000d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.259+0000 7faa04f25640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0015e770 con 0x7faa000b3320 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.259+0000 7faa04f25640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7faa000b3190 con 0x7faa0015e390 2024-03-24T10:35:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.259+0000 7faa04f25640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faa000d4e40 con 0x7faa000d1000 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa000b3320 0x7faa000cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33340/0 (socket says 172.21.15.1:33340) 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fed9d640 1 -- 172.21.15.1:0/4002157262 learned_addr learned my addr 172.21.15.1:0/4002157262 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/4002157262 >> v2:172.21.15.3:3300/0 conn(0x7faa0015e390 0x7faa000d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7faa000d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244559465 0 0) 0x7fa9f8001490 con 0x7faa000b3320 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9ec003520 con 0x7faa000b3320 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3816298949 0 0) 0x7fa9ec003520 con 0x7faa000b3320 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 msgr2=0x7faa000d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7faa000d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 >> v2:172.21.15.3:3300/0 conn(0x7faa0015e390 msgr2=0x7faa000d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 --2- 172.21.15.1:0/4002157262 >> v2:172.21.15.3:3300/0 conn(0x7faa0015e390 0x7faa000d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000d1e40 con 0x7faa000b3320 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fa9f8001b80 con 0x7faa000b3320 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/4002157262 >> v2:172.21.15.3:3300/0 conn(0x7faa0015e390 0x7faa000d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 >> v1:172.21.15.1:6789/0 conn(0x7faa000b3320 legacy=0x7faa000cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7fa9ec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ec003ef0 0x7fa9ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7fa9ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa000d4e40 con 0x7fa9ec004d00 2024-03-24T10:35:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec00b9e0 con 0x7fa9ec003ef0 2024-03-24T10:35:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ec00bba0 con 0x7faa000d1000 2024-03-24T10:35:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fed9d640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7fa9ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.260+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ec003ef0 0x7fa9ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7fa9ec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fed9d640 1 -- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 msgr2=0x7fa9ec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fed9d640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa000d1000 0x7fa9ec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fed9d640 1 -- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ec003ef0 msgr2=0x7fa9ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fed9d640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ec003ef0 0x7fa9ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.261+0000 7fa9fed9d640 1 -- 172.21.15.1:0/4002157262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec003520 con 0x7fa9ec004d00 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.262+0000 7fa9fed9d640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7fa9ec005180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa000b37d0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8262178b62694076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.262+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8034630 con 0x7fa9ec004d00 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.262+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f802f070 con 0x7fa9ec004d00 2024-03-24T10:35:23.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.262+0000 7fa9fd59a640 1 -- 172.21.15.1:0/4002157262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8034a30 con 0x7fa9ec004d00 2024-03-24T10:35:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 -- 172.21.15.1:0/4002157262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 msgr2=0x7fa9ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 --2- 172.21.15.1:0/4002157262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7fa9ec005180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa000b37d0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7fa9fed9d640 1 -- 172.21.15.1:0/4002157262 reap_dead start 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 -- 172.21.15.1:0/4002157262 shutdown_connections 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 -- 172.21.15.1:0/4002157262 >> 172.21.15.1:0/4002157262 conn(0x7faa000d3ec0 msgr2=0x7faa000d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 -- 172.21.15.1:0/4002157262 shutdown_connections 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.263+0000 7faa04f25640 1 -- 172.21.15.1:0/4002157262 wait complete. 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.264+0000 7faa04f25640 1 Processor -- start 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.264+0000 7faa04f25640 1 -- start start 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 0x7faa00150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa0015aec0 0x7faa0015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa001f6c90 con 0x7fa9ec004d00 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7faa0015b880 con 0x7faa001542e0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7faa04f25640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faa00150bd0 con 0x7faa0015aec0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49168/0 (socket says 172.21.15.1:49168) 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 -- 172.21.15.1:0/776182509 learned_addr learned my addr 172.21.15.1:0/776182509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fed9d640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa0015aec0 0x7faa0015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 0x7faa00150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 -- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa0015aec0 msgr2=0x7faa0015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa0015aec0 0x7faa0015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 -- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 msgr2=0x7faa00150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 0x7faa00150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 -- 172.21.15.1:0/776182509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa00150eb0 con 0x7fa9ec004d00 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 0x7faa00150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.265+0000 7fa9fe59c640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0011cd0 tx=0x7fa9f000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.266+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000fcc0 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.266+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f000fe40 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.266+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0004260 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.266+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa00151190 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.266+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0014e5c0 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.267+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0004f4e0 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.267+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9f00043c0 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.267+0000 7fa9f6ffd640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 0x7fa9e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.267+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fa9f008aac0 con 0x7fa9ec004d00 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.267+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 0x7fa9e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.268+0000 7fa9fdd9b640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 0x7fa9e006dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faa00154950 tx=0x7fa9e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.272+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9f00553e0 con 0x7fa9ec004d00 2024-03-24T10:35:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.454+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa000d2280 con 0x7fa9ec004d00 2024-03-24T10:35:23.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.456+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/776182509 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7fa9f008ae60 con 0x7fa9ec004d00 2024-03-24T10:35:23.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:23.457 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 msgr2=0x7fa9e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 0x7fa9e006dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faa00154950 tx=0x7fa9e8008040 comp rx=0 tx=0).stop 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 msgr2=0x7faa001f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0011cd0 tx=0x7fa9f000f610 comp rx=0 tx=0).stop 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 shutdown_connections 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa9e006b810 0x7fa9e006dcd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faa0015aec0 0x7faa0015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa001542e0 0x7faa00150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 --2- 172.21.15.1:0/776182509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9ec004d00 0x7faa001f6750 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 >> 172.21.15.1:0/776182509 conn(0x7faa000d3ec0 msgr2=0x7faa000d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 shutdown_connections 2024-03-24T10:35:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.458+0000 7faa04f25640 1 -- 172.21.15.1:0/776182509 wait complete. 2024-03-24T10:35:23.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:23.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.607+0000 7ff04d82a640 1 Processor -- start 2024-03-24T10:35:23.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.607+0000 7ff04d82a640 1 -- start start 2024-03-24T10:35:23.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff04d82a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0480d2a20 0x7ff0480cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff04d82a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0480b2310 0x7ff0480cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff04d82a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0480b0b70 con 0x7ff04815e390 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff04d82a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff0480d1d60 con 0x7ff0480d2a20 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff04d82a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff0480b0de0 con 0x7ff0480b2310 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0480d2a20 0x7ff0480cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0480d2a20 0x7ff0480cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49174/0 (socket says 172.21.15.1:49174) 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 -- 172.21.15.1:0/4279000838 learned_addr learned my addr 172.21.15.1:0/4279000838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0477fe640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0480b2310 0x7ff0480cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 -- 172.21.15.1:0/4279000838 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0480b2310 msgr2=0x7ff0480cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0480b2310 0x7ff0480cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 -- 172.21.15.1:0/4279000838 >> v1:172.21.15.3:6789/0 conn(0x7ff04815e390 legacy=0x7ff0480d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.608+0000 7ff0467fc640 1 -- 172.21.15.1:0/4279000838 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04815e770 con 0x7ff0480d2a20 2024-03-24T10:35:23.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff0467fc640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480cead0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff030009870 tx=0x7ff03002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a260c02e74ed90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff045ffb640 1 -- 172.21.15.1:0/4279000838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030031640 con 0x7ff0480d2a20 2024-03-24T10:35:23.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff045ffb640 1 -- 172.21.15.1:0/4279000838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0300317a0 con 0x7ff0480d2a20 2024-03-24T10:35:23.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff045ffb640 1 -- 172.21.15.1:0/4279000838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030031b30 con 0x7ff0480d2a20 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff04d82a640 1 -- 172.21.15.1:0/4279000838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 msgr2=0x7ff0480cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.609+0000 7ff04d82a640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480cead0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff030009870 tx=0x7ff03002b8f0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 -- 172.21.15.1:0/4279000838 shutdown_connections 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff0480b2310 0x7ff0480cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 --2- 172.21.15.1:0/4279000838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480cead0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 -- 172.21.15.1:0/4279000838 >> 172.21.15.1:0/4279000838 conn(0x7ff0480d3ee0 msgr2=0x7ff0480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 -- 172.21.15.1:0/4279000838 shutdown_connections 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.610+0000 7ff04d82a640 1 -- 172.21.15.1:0/4279000838 wait complete. 2024-03-24T10:35:23.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.611+0000 7ff04d82a640 1 Processor -- start 2024-03-24T10:35:23.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.611+0000 7ff04d82a640 1 -- start start 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.612+0000 7ff04d82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0480b2310 0x7ff0480cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.612+0000 7ff04d82a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.612+0000 7ff04d82a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff04815e390 0x7ff0480b9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff04d82a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0480cd120 con 0x7ff0480d2a20 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff04d82a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff0480cd400 con 0x7ff0480b2310 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff04d82a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff0480cd6e0 con 0x7ff04815e390 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49176/0 (socket says 172.21.15.1:49176) 2024-03-24T10:35:23.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 -- 172.21.15.1:0/3442397422 learned_addr learned my addr 172.21.15.1:0/3442397422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff046ffd640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0480b2310 0x7ff0480cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0477fe640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff04815e390 0x7ff0480b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 -- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff04815e390 msgr2=0x7ff0480b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff04815e390 0x7ff0480b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 -- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0480b2310 msgr2=0x7ff0480cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0480b2310 0x7ff0480cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 -- 172.21.15.1:0/3442397422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0300094d0 con 0x7ff0480d2a20 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff0467fc640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff030009870 tx=0x7ff0300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03003ac60 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03003ade0 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030044630 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0480cd9c0 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.613+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0480ccc60 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.614+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0480cdee0 con 0x7ff0480d2a20 2024-03-24T10:35:23.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.615+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff03002f050 con 0x7ff0480d2a20 2024-03-24T10:35:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.615+0000 7ff03ffff640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 0x7ff02806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.616+0000 7ff046ffd640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 0x7ff02806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.616+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7ff0300ac4a0 con 0x7ff0480d2a20 2024-03-24T10:35:23.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.616+0000 7ff046ffd640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 0x7ff02806dd00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff038009f20 tx=0x7ff0380094d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.620+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff0480cdee0 con 0x7ff0480d2a20 2024-03-24T10:35:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.800+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff0480cdee0 con 0x7ff0480d2a20 2024-03-24T10:35:23.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.800+0000 7ff03ffff640 1 -- 172.21.15.1:0/3442397422 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7ff0480cdee0 con 0x7ff0480d2a20 2024-03-24T10:35:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 msgr2=0x7ff02806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 0x7ff02806dd00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff038009f20 tx=0x7ff0380094d0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 msgr2=0x7ff0480b9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff030009870 tx=0x7ff0300046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 shutdown_connections 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff02806b840 0x7ff02806dd00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff04815e390 0x7ff0480b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0480d2a20 0x7ff0480b9010 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 --2- 172.21.15.1:0/3442397422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0480b2310 0x7ff0480cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.802+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 >> 172.21.15.1:0/3442397422 conn(0x7ff0480d3ee0 msgr2=0x7ff04815e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.803+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 shutdown_connections 2024-03-24T10:35:23.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.803+0000 7ff04d82a640 1 -- 172.21.15.1:0/3442397422 wait complete. 2024-03-24T10:35:23.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:23.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.917+0000 7f73a90ae640 1 Processor -- start 2024-03-24T10:35:23.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 -- start start 2024-03-24T10:35:23.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73a41615f0 0x7f73a40d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a40b4e40 0x7f73a4161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 0x7f73a4162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f73a40cfd50 con 0x7f73a41615f0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f73a40c7a40 con 0x7f73a41619d0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a90ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f73a40b55b0 con 0x7f73a40b4e40 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73a41615f0 0x7f73a40d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f73a41615f0 0x7f73a40d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49190/0 (socket says 172.21.15.1:49190) 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.918+0000 7f73a2d9d640 1 -- 172.21.15.1:0/1132381229 learned_addr learned my addr 172.21.15.1:0/1132381229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a359e640 1 --2- 172.21.15.1:0/1132381229 >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 0x7f73a4162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f739bfff640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a40b4e40 0x7f73a4161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 -- 172.21.15.1:0/1132381229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a40b4e40 msgr2=0x7f73a4161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a40b4e40 0x7f73a4161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 -- 172.21.15.1:0/1132381229 >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 msgr2=0x7f73a4162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/1132381229 >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 0x7f73a4162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 -- 172.21.15.1:0/1132381229 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a40b52d0 con 0x7f73a41615f0 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a359e640 1 --2- 172.21.15.1:0/1132381229 >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 0x7f73a4162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:23.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.919+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41615f0 0x7f73a40d4a10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f73a4073780 tx=0x7f739402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be282b4a85c67347 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a259c640 1 -- 172.21.15.1:0/1132381229 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394031870 con 0x7f73a41615f0 2024-03-24T10:35:23.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a259c640 1 -- 172.21.15.1:0/1132381229 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73940319d0 con 0x7f73a41615f0 2024-03-24T10:35:23.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a259c640 1 -- 172.21.15.1:0/1132381229 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394031d60 con 0x7f73a41615f0 2024-03-24T10:35:23.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 -- 172.21.15.1:0/1132381229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41615f0 msgr2=0x7f73a40d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41615f0 0x7f73a40d4a10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f73a4073780 tx=0x7f739402b910 comp rx=0 tx=0).stop 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 -- 172.21.15.1:0/1132381229 shutdown_connections 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 --2- 172.21.15.1:0/1132381229 >> v2:172.21.15.3:3300/0 conn(0x7f73a41619d0 0x7f73a4162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a40b4e40 0x7f73a4161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 --2- 172.21.15.1:0/1132381229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41615f0 0x7f73a40d4a10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 -- 172.21.15.1:0/1132381229 >> 172.21.15.1:0/1132381229 conn(0x7f73a40d3ef0 msgr2=0x7f73a40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 -- 172.21.15.1:0/1132381229 shutdown_connections 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.920+0000 7f73a90ae640 1 -- 172.21.15.1:0/1132381229 wait complete. 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.922+0000 7f73a90ae640 1 Processor -- start 2024-03-24T10:35:23.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.922+0000 7f73a90ae640 1 -- start start 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a41615f0 0x7f73a415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73a4154ce0 0x7f73a406c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f73a40b84b0 con 0x7f73a41619d0 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f73a4157200 con 0x7f73a4154ce0 2024-03-24T10:35:23.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.923+0000 7f73a90ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f73a40b7df0 con 0x7f73a41615f0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49206/0 (socket says 172.21.15.1:49206) 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 -- 172.21.15.1:0/111825647 learned_addr learned my addr 172.21.15.1:0/111825647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a41615f0 0x7f73a415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f73a359e640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73a4154ce0 0x7f73a406c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 -- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a41615f0 msgr2=0x7f73a415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a41615f0 0x7f73a415c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 -- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73a4154ce0 msgr2=0x7f73a406c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73a4154ce0 0x7f73a406c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 -- 172.21.15.1:0/111825647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73940094d0 con 0x7f73a41619d0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739bfff640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f738c00c7f0 tx=0x7f738c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c004220 con 0x7f73a41619d0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738c004380 con 0x7f73a41619d0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a40b80d0 con 0x7f73a41619d0 2024-03-24T10:35:23.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c007860 con 0x7f73a41619d0 2024-03-24T10:35:23.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.924+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a41574e0 con 0x7f73a41619d0 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.926+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a40b85f0 con 0x7f73a41619d0 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.926+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f738c007bd0 con 0x7f73a41619d0 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.926+0000 7f739b7fe640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 0x7f738406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.926+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f738c08a3d0 con 0x7f73a41619d0 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.926+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 0x7f738406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:23.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.927+0000 7f73a2d9d640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 0x7f738406dcd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7394036000 tx=0x7f739403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:23.930+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73a40b85f0 con 0x7f73a41619d0 2024-03-24T10:35:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.109+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f73a40b85f0 con 0x7f73a41619d0 2024-03-24T10:35:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.109+0000 7f739b7fe640 1 -- 172.21.15.1:0/111825647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f73a40b85f0 con 0x7f73a41619d0 2024-03-24T10:35:24.111 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 msgr2=0x7f738406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 0x7f738406dcd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7394036000 tx=0x7f739403c040 comp rx=0 tx=0).stop 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 msgr2=0x7f73a41569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f738c00c7f0 tx=0x7f738c00ccb0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 shutdown_connections 2024-03-24T10:35:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f738406b810 0x7f738406dcd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73a4154ce0 0x7f73a406c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73a41619d0 0x7f73a41569d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 --2- 172.21.15.1:0/111825647 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f73a41615f0 0x7f73a415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 >> 172.21.15.1:0/111825647 conn(0x7f73a40d3ef0 msgr2=0x7f73a415f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 shutdown_connections 2024-03-24T10:35:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.111+0000 7f73a90ae640 1 -- 172.21.15.1:0/111825647 wait complete. 2024-03-24T10:35:24.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:24.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.232+0000 7f6f6fec3640 1 Processor -- start 2024-03-24T10:35:24.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.232+0000 7f6f6fec3640 1 -- start start 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6fec3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f68161f60 0x7f6f680d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6fec3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f6806c3c0 0x7f6f6815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6fec3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f680d1a50 con 0x7f6f68162340 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6fec3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6f6806c850 con 0x7f6f68161f60 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6fec3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f6f680cd290 con 0x7f6f6806c3c0 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f68161f60 0x7f6f680d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6d45e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f68162340 0x7f6f6815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33348/0 (socket says 172.21.15.1:33348) 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f68161f60 0x7f6f680d2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49222/0 (socket says 172.21.15.1:49222) 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6d45e640 1 -- 172.21.15.1:0/3598741024 learned_addr learned my addr 172.21.15.1:0/3598741024 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6e460640 1 --2- 172.21.15.1:0/3598741024 >> v2:172.21.15.3:3300/0 conn(0x7f6f6806c3c0 0x7f6f6815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 -- 172.21.15.1:0/3598741024 >> v2:172.21.15.3:3300/0 conn(0x7f6f6806c3c0 msgr2=0x7f6f6815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 --2- 172.21.15.1:0/3598741024 >> v2:172.21.15.3:3300/0 conn(0x7f6f6806c3c0 0x7f6f6815e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 -- 172.21.15.1:0/3598741024 >> v1:172.21.15.1:6789/0 conn(0x7f6f68162340 legacy=0x7f6f6815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6dc5f640 1 -- 172.21.15.1:0/3598741024 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f680cff20 con 0x7f6f68161f60 2024-03-24T10:35:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.233+0000 7f6f6cc5d640 1 -- 172.21.15.1:0/3598741024 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030158569 0 0) 0x7f6f680d1a50 con 0x7f6f68162340 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.234+0000 7f6f6dc5f640 1 --2- 172.21.15.1:0/3598741024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f680d2750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f680b36d0 tx=0x7f6f6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff52ff54c6aa259b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.234+0000 7f6f6cc5d640 1 -- 172.21.15.1:0/3598741024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64031830 con 0x7f6f68161f60 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.234+0000 7f6f6cc5d640 1 -- 172.21.15.1:0/3598741024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64031990 con 0x7f6f68161f60 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.234+0000 7f6f6cc5d640 1 -- 172.21.15.1:0/3598741024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64031d20 con 0x7f6f68161f60 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 -- 172.21.15.1:0/3598741024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 msgr2=0x7f6f680d2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/3598741024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f680d2750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f680b36d0 tx=0x7f6f6402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 -- 172.21.15.1:0/3598741024 shutdown_connections 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/3598741024 >> v2:172.21.15.3:3300/0 conn(0x7f6f6806c3c0 0x7f6f6815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/3598741024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f680d2750 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 -- 172.21.15.1:0/3598741024 >> 172.21.15.1:0/3598741024 conn(0x7f6f680d1c30 msgr2=0x7f6f680d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 -- 172.21.15.1:0/3598741024 shutdown_connections 2024-03-24T10:35:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.235+0000 7f6f6fec3640 1 -- 172.21.15.1:0/3598741024 wait complete. 2024-03-24T10:35:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.237+0000 7f6f6fec3640 1 Processor -- start 2024-03-24T10:35:24.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.237+0000 7f6f6fec3640 1 -- start start 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f6806c3c0 0x7f6f680c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f68162340 0x7f6f68156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6806fb00 con 0x7f6f68161f60 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6f680c7e30 con 0x7f6f68162340 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6fec3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6f680cb4b0 con 0x7f6f6806c3c0 2024-03-24T10:35:24.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49226/0 (socket says 172.21.15.1:49226) 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 -- 172.21.15.1:0/1865604820 learned_addr learned my addr 172.21.15.1:0/1865604820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6e460640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f68162340 0x7f6f68156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6dc5f640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f6806c3c0 0x7f6f680c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 -- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f6806c3c0 msgr2=0x7f6f680c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f6806c3c0 0x7f6f680c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 -- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f68162340 msgr2=0x7f6f68156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.238+0000 7f6f6d45e640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f68162340 0x7f6f68156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f6d45e640 1 -- 172.21.15.1:0/1865604820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f64009400 con 0x7f6f68161f60 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f6d45e640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c4a0 tx=0x7f6f5800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f580074e0 con 0x7f6f68161f60 2024-03-24T10:35:24.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f58007640 con 0x7f6f68161f60 2024-03-24T10:35:24.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f680cb790 con 0x7f6f68161f60 2024-03-24T10:35:24.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58004a00 con 0x7f6f68161f60 2024-03-24T10:35:24.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.239+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f680b4190 con 0x7f6f68161f60 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.240+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f680b4380 con 0x7f6f68161f60 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.240+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f58004b60 con 0x7f6f68161f60 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.241+0000 7f6f5e7fc640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 0x7f6f4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.241+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f6f58089810 con 0x7f6f68161f60 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.241+0000 7f6f6dc5f640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 0x7f6f4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.241+0000 7f6f6dc5f640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 0x7f6f4c06dcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f64036000 tx=0x7f6f6403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.245+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f58054150 con 0x7f6f68161f60 2024-03-24T10:35:24.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.506+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f680cbb30 con 0x7f6f68161f60 2024-03-24T10:35:24.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.506+0000 7f6f5e7fc640 1 -- 172.21.15.1:0/1865604820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7f6f58057810 con 0x7f6f68161f60 2024-03-24T10:35:24.507 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 msgr2=0x7f6f4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 0x7f6f4c06dcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f64036000 tx=0x7f6f6403c040 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 msgr2=0x7f6f681f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800c4a0 tx=0x7f6f5800c960 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 shutdown_connections 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6f4c06b810 0x7f6f4c06dcd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f68162340 0x7f6f68156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f68161f60 0x7f6f681f3ad0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 --2- 172.21.15.1:0/1865604820 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6f6806c3c0 0x7f6f680c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 >> 172.21.15.1:0/1865604820 conn(0x7f6f680d1c30 msgr2=0x7f6f680b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.508+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 shutdown_connections 2024-03-24T10:35:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.509+0000 7f6f6fec3640 1 -- 172.21.15.1:0/1865604820 wait complete. 2024-03-24T10:35:24.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:24.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.625+0000 7f455c1e4640 1 Processor -- start 2024-03-24T10:35:24.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.625+0000 7f455c1e4640 1 -- start start 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455c1e4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f45540d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455c1e4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45540d2930 0x7f45540cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455c1e4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45540d0e30 con 0x7f45540cd970 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455c1e4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f45540cf8e0 con 0x7f45540d2930 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455c1e4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f45540cfb30 con 0x7f455415c1e0 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455a781640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f45540cd970 0x7f45540ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33354/0 (socket says 172.21.15.1:33354) 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.626+0000 7f455a781640 1 -- 172.21.15.1:0/3982616681 learned_addr learned my addr 172.21.15.1:0/3982616681 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:24.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4559f80640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f45540d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 --2- 172.21.15.1:0/3982616681 >> v2:172.21.15.3:3300/0 conn(0x7f45540d2930 0x7f45540cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483137606 0 0) 0x7f454c001490 con 0x7f45540cd970 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4540003520 con 0x7f45540cd970 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674466392 0 0) 0x7f4540003520 con 0x7f45540cd970 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 msgr2=0x7f45540d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f45540d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 >> v2:172.21.15.3:3300/0 conn(0x7f45540d2930 msgr2=0x7f45540cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 --2- 172.21.15.1:0/3982616681 >> v2:172.21.15.3:3300/0 conn(0x7f45540d2930 0x7f45540cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45540cfe10 con 0x7f45540cd970 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f454c001b80 con 0x7f45540cd970 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 >> v1:172.21.15.1:6789/0 conn(0x7f45540cd970 legacy=0x7f45540ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f4540003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4540003ef0 0x7f45400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540004d00 0x7f4540005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f45540cf8e0 con 0x7f4540003ef0 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f45540cfb30 con 0x7f4540004d00 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f454000b9b0 con 0x7f455415c1e0 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4540003ef0 0x7f45400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f4559f80640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f4540003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455a781640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540004d00 0x7f4540005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 -- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 msgr2=0x7f4540003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455415c1e0 0x7f4540003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 -- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540004d00 msgr2=0x7f4540005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540004d00 0x7f4540005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.627+0000 7f455977f640 1 -- 172.21.15.1:0/3982616681 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4540003520 con 0x7f4540003ef0 2024-03-24T10:35:24.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455977f640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4540003ef0 0x7f45400047c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4544009ac0 tx=0x7f45400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed13f20fbca5ac4c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454402c710 con 0x7f4540003ef0 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454402c870 con 0x7f4540003ef0 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f4558f7e640 1 -- 172.21.15.1:0/3982616681 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454402cc00 con 0x7f4540003ef0 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 -- 172.21.15.1:0/3982616681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4540003ef0 msgr2=0x7f45400047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3982616681 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4540003ef0 0x7f45400047c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4544009ac0 tx=0x7f45400057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455a781640 1 -- 172.21.15.1:0/3982616681 reap_dead start 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 -- 172.21.15.1:0/3982616681 shutdown_connections 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 -- 172.21.15.1:0/3982616681 >> 172.21.15.1:0/3982616681 conn(0x7f45540d1d10 msgr2=0x7f45540d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 -- 172.21.15.1:0/3982616681 shutdown_connections 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.629+0000 7f455c1e4640 1 -- 172.21.15.1:0/3982616681 wait complete. 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.630+0000 7f455c1e4640 1 Processor -- start 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 -- start start 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540003ef0 0x7f455415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 0x7f455409e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f455409f7a0 con 0x7f45540ca620 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f45540cafe0 con 0x7f4540003ef0 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455c1e4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f455409fbc0 con 0x7f455409cca0 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455a781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455a781640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49244/0 (socket says 172.21.15.1:49244) 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.631+0000 7f455a781640 1 -- 172.21.15.1:0/3014847101 learned_addr learned my addr 172.21.15.1:0/3014847101 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f4559f80640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540003ef0 0x7f455415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455977f640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 0x7f455409e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 -- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 msgr2=0x7f455409e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 0x7f455409e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 -- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540003ef0 msgr2=0x7f455415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540003ef0 0x7f455415a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 -- 172.21.15.1:0/3014847101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455409eb20 con 0x7f45540ca620 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455977f640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 0x7f455409e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455a781640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f454c000f80 tx=0x7f454c00af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c0078c0 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454c007a20 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455409ee00 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c013630 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.632+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455409f3c0 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.633+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455409f580 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.633+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f454c013790 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.634+0000 7f454a7fc640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 0x7f453806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.634+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f454c08a810 con 0x7f45540ca620 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.634+0000 7f4559f80640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 0x7f453806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.634+0000 7f4559f80640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 0x7f453806dd00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4550009a20 tx=0x7f4550008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.638+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f454c055130 con 0x7f45540ca620 2024-03-24T10:35:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.818+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f45540d0e30 con 0x7f45540ca620 2024-03-24T10:35:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.818+0000 7f454a7fc640 1 -- 172.21.15.1:0/3014847101 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f45540d0e30 con 0x7f45540ca620 2024-03-24T10:35:24.819 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:24.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 msgr2=0x7f453806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 0x7f453806dd00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4550009a20 tx=0x7f4550008040 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 msgr2=0x7f45540caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f454c000f80 tx=0x7f454c00af80 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 shutdown_connections 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f453806b840 0x7f453806dd00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45540ca620 0x7f45540caaa0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f455409cca0 0x7f455409e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 --2- 172.21.15.1:0/3014847101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4540003ef0 0x7f455415a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 >> 172.21.15.1:0/3014847101 conn(0x7f45540d1d10 msgr2=0x7f455415f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 shutdown_connections 2024-03-24T10:35:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.820+0000 7f455c1e4640 1 -- 172.21.15.1:0/3014847101 wait complete. 2024-03-24T10:35:24.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.935+0000 7fdcfa16c640 1 Processor -- start 2024-03-24T10:35:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.935+0000 7fdcfa16c640 1 -- start start 2024-03-24T10:35:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcf415e390 0x7fdcf40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b3320 0x7fdcf40cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf40b3190 con 0x7fdcf415e390 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf415e770 con 0x7fdcf40b3320 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcfa16c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf40d4e40 con 0x7fdcf40d1000 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcf37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcf415e390 0x7fdcf40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcf37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcf415e390 0x7fdcf40d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49256/0 (socket says 172.21.15.1:49256) 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3254975958 learned_addr learned my addr 172.21.15.1:0/3254975958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcf3fff640 1 --2- 172.21.15.1:0/3254975958 >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b3320 0x7fdcf40cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.936+0000 7fdcf2ffd640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3254975958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 msgr2=0x7fdcf40d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3254975958 >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b3320 msgr2=0x7fdcf40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/3254975958 >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b3320 0x7fdcf40cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 -- 172.21.15.1:0/3254975958 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf40d1e40 con 0x7fdcf415e390 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf2ffd640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:24.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf415e390 0x7fdcf40d24a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b5960 tx=0x7fdce402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26709adebbcd4a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.937+0000 7fdcf27fc640 1 -- 172.21.15.1:0/3254975958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4031750 con 0x7fdcf415e390 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcf27fc640 1 -- 172.21.15.1:0/3254975958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce40318b0 con 0x7fdcf415e390 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcf27fc640 1 -- 172.21.15.1:0/3254975958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4031c40 con 0x7fdcf415e390 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 -- 172.21.15.1:0/3254975958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf415e390 msgr2=0x7fdcf40d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf415e390 0x7fdcf40d24a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b5960 tx=0x7fdce402b7f0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 -- 172.21.15.1:0/3254975958 shutdown_connections 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/3254975958 >> v2:172.21.15.3:3300/0 conn(0x7fdcf40b3320 0x7fdcf40cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/3254975958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf415e390 0x7fdcf40d24a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 -- 172.21.15.1:0/3254975958 >> 172.21.15.1:0/3254975958 conn(0x7fdcf40d3ec0 msgr2=0x7fdcf40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 -- 172.21.15.1:0/3254975958 shutdown_connections 2024-03-24T10:35:24.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.938+0000 7fdcfa16c640 1 -- 172.21.15.1:0/3254975958 wait complete. 2024-03-24T10:35:24.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.939+0000 7fdcfa16c640 1 Processor -- start 2024-03-24T10:35:24.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.939+0000 7fdcfa16c640 1 -- start start 2024-03-24T10:35:24.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 0x7fdcf40b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf409e240 con 0x7fdcf40b3320 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf409dc10 con 0x7fdcf415e390 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcfa16c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf40cca30 con 0x7fdcf40d1000 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49270/0 (socket says 172.21.15.1:49270) 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.940+0000 7fdcf37fe640 1 -- 172.21.15.1:0/2512595391 learned_addr learned my addr 172.21.15.1:0/2512595391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf3fff640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 0x7fdcf40b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf2ffd640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 -- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 msgr2=0x7fdcf40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 -- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 msgr2=0x7fdcf40b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 0x7fdcf40b8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 -- 172.21.15.1:0/2512595391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce40094d0 con 0x7fdcf40b3320 2024-03-24T10:35:24.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf3fff640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 0x7fdcf40b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcf37fe640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce40339d0 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce4033b30 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf40ccd10 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce403c400 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.941+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf40cd730 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.942+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf409e380 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.943+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdce403a040 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.943+0000 7fdcdbfff640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 0x7fdce006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.943+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fdce40abdf0 con 0x7fdcf40b3320 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.943+0000 7fdcf2ffd640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 0x7fdce006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:24.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.943+0000 7fdcf2ffd640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 0x7fdce006dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b9600 tx=0x7fdce8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:24.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:24.947+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdce4076710 con 0x7fdcf40b3320 2024-03-24T10:35:25.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.140+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdcf404f4e0 con 0x7fdcf40b3320 2024-03-24T10:35:25.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.140+0000 7fdcdbfff640 1 -- 172.21.15.1:0/2512595391 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7fdce4038080 con 0x7fdcf40b3320 2024-03-24T10:35:25.141 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 msgr2=0x7fdce006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 0x7fdce006dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b9600 tx=0x7fdce8008040 comp rx=0 tx=0).stop 2024-03-24T10:35:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 msgr2=0x7fdcf414f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce40046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 shutdown_connections 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdce006b810 0x7fdce006dcd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcf415e390 0x7fdcf40b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdcf40d1000 0x7fdcf40cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 --2- 172.21.15.1:0/2512595391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcf40b3320 0x7fdcf414f1c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 >> 172.21.15.1:0/2512595391 conn(0x7fdcf40d3ec0 msgr2=0x7fdcf40cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 shutdown_connections 2024-03-24T10:35:25.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.142+0000 7fdcfa16c640 1 -- 172.21.15.1:0/2512595391 wait complete. 2024-03-24T10:35:25.155 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:25.155 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:25.170 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.312+0000 7f7c4abf4640 1 Processor -- start 2024-03-24T10:35:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.312+0000 7f7c4abf4640 1 -- start start 2024-03-24T10:35:25.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c4abf4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c440d2bb0 0x7f7c440b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c4abf4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d1000 0x7f7c440b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c4abf4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4415e770 con 0x7f7c4415e390 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c4abf4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7c440b3190 con 0x7f7c440d2bb0 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c4abf4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7c440d4e40 con 0x7f7c440d1000 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c43fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c440d2bb0 0x7f7c440b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c43fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c440d2bb0 0x7f7c440b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49284/0 (socket says 172.21.15.1:49284) 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c43fff640 1 -- 172.21.15.1:0/3754314979 learned_addr learned my addr 172.21.15.1:0/3754314979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.313+0000 7f7c49191640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d1000 0x7f7c440b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.314+0000 7f7c43fff640 1 -- 172.21.15.1:0/3754314979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d1000 msgr2=0x7f7c440b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.314+0000 7f7c43fff640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d1000 0x7f7c440b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.314+0000 7f7c43fff640 1 -- 172.21.15.1:0/3754314979 >> v1:172.21.15.3:6789/0 conn(0x7f7c4415e390 legacy=0x7f7c440d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.314+0000 7f7c43fff640 1 -- 172.21.15.1:0/3754314979 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c440d1e40 con 0x7f7c440d2bb0 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.314+0000 7f7c43fff640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c440d2bb0 0x7f7c440b3b90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c4e4f2821612ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c437fe640 1 -- 172.21.15.1:0/3754314979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c380317c0 con 0x7f7c440d2bb0 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c437fe640 1 -- 172.21.15.1:0/3754314979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c38031920 con 0x7f7c440d2bb0 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c437fe640 1 -- 172.21.15.1:0/3754314979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38031cb0 con 0x7f7c440d2bb0 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 -- 172.21.15.1:0/3754314979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c440d2bb0 msgr2=0x7f7c440b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c440d2bb0 0x7f7c440b3b90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802b860 comp rx=0 tx=0).stop 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 -- 172.21.15.1:0/3754314979 shutdown_connections 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d1000 0x7f7c440b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/3754314979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c440d2bb0 0x7f7c440b3b90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 -- 172.21.15.1:0/3754314979 >> 172.21.15.1:0/3754314979 conn(0x7f7c440d3ec0 msgr2=0x7f7c440d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 -- 172.21.15.1:0/3754314979 shutdown_connections 2024-03-24T10:35:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.315+0000 7f7c4abf4640 1 -- 172.21.15.1:0/3754314979 wait complete. 2024-03-24T10:35:25.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.317+0000 7f7c4abf4640 1 Processor -- start 2024-03-24T10:35:25.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.317+0000 7f7c4abf4640 1 -- start start 2024-03-24T10:35:25.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c440d1000 0x7f7c441543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d2bb0 0x7f7c4409ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7c44154910 con 0x7f7c4415e390 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4409f2d0 con 0x7f7c440d1000 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.318+0000 7f7c4abf4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7c440b8de0 con 0x7f7c440d2bb0 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49292/0 (socket says 172.21.15.1:49292) 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 -- 172.21.15.1:0/2885679553 learned_addr learned my addr 172.21.15.1:0/2885679553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c48990640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c440d1000 0x7f7c441543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c43fff640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d2bb0 0x7f7c4409ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 -- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d2bb0 msgr2=0x7f7c4409ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d2bb0 0x7f7c4409ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 -- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c440d1000 msgr2=0x7f7c441543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c440d1000 0x7f7c441543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 -- 172.21.15.1:0/2885679553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c380094d0 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c49191640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00efc0 tx=0x7f7c3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c009ca0 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c00f040 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0146c0 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c440b8820 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.319+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c440cb5d0 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.320+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4404f4e0 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.321+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7c3c014820 con 0x7f7c4415e390 2024-03-24T10:35:25.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.321+0000 7f7c417fa640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.322+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f7c3c08b140 con 0x7f7c4415e390 2024-03-24T10:35:25.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.322+0000 7f7c48990640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.322+0000 7f7c48990640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 0x7f7c3006dcd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ce350 tx=0x7f7c34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.326+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c3c055ab0 con 0x7f7c4415e390 2024-03-24T10:35:25.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.510+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c440d2280 con 0x7f7c4415e390 2024-03-24T10:35:25.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.512+0000 7f7c417fa640 1 -- 172.21.15.1:0/2885679553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f7c3c059170 con 0x7f7c4415e390 2024-03-24T10:35:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:25.513 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.513+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 msgr2=0x7f7c3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.513+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 0x7f7c3006dcd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ce350 tx=0x7f7c34009040 comp rx=0 tx=0).stop 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.513+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 msgr2=0x7f7c4406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.513+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00efc0 tx=0x7f7c3c00c490 comp rx=0 tx=0).stop 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 shutdown_connections 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c4415e390 0x7f7c4406c760 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c440d2bb0 0x7f7c4409ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 --2- 172.21.15.1:0/2885679553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c440d1000 0x7f7c441543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 >> 172.21.15.1:0/2885679553 conn(0x7f7c440d3ec0 msgr2=0x7f7c4406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 shutdown_connections 2024-03-24T10:35:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.514+0000 7f7c4abf4640 1 -- 172.21.15.1:0/2885679553 wait complete. 2024-03-24T10:35:25.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:25.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.633+0000 7f369f6c3640 1 Processor -- start 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.634+0000 7f369f6c3640 1 -- start start 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.634+0000 7f369f6c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36980d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.634+0000 7f369f6c3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36980b0b70 con 0x7f36980cfae0 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.634+0000 7f369f6c3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36980d1d60 con 0x7f36980b2ed0 2024-03-24T10:35:25.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.634+0000 7f369f6c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f36980b0de0 con 0x7f369815e390 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369dc60640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f36980cfae0 0x7f36980b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33362/0 (socket says 172.21.15.1:33362) 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369dc60640 1 -- 172.21.15.1:0/60883900 learned_addr learned my addr 172.21.15.1:0/60883900 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369d45f640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36980d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344929207 0 0) 0x7f36980b0b70 con 0x7f36980cfae0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3684003520 con 0x7f36980cfae0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 317847451 0 0) 0x7f3684003520 con 0x7f36980cfae0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 msgr2=0x7f36980d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36980d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 >> v1:172.21.15.3:6789/0 conn(0x7f36980b2ed0 legacy=0x7f36980b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369815e770 con 0x7f36980cfae0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f36900019f0 con 0x7f36980cfae0 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 >> v1:172.21.15.1:6789/0 conn(0x7f36980cfae0 legacy=0x7f36980b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36840039a0 0x7f3684003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3684004d00 0x7f3684005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f36980b0de0 con 0x7f3684004d00 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f368400b980 con 0x7f36840039a0 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f368400bb40 con 0x7f369815e390 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369dc60640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3684004d00 0x7f3684005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369d45f640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36840039a0 0x7f3684003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.635+0000 7f369cc5e640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 -- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 msgr2=0x7f36840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f369815e390 0x7f36840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 -- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36840039a0 msgr2=0x7f3684003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36840039a0 0x7f3684003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 -- 172.21.15.1:0/60883900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3684003520 con 0x7f3684004d00 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.636+0000 7f369dc60640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3684004d00 0x7f3684005180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f369000a7d0 tx=0x7f36840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b33ea05cdea412 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690034630 con 0x7f3684004d00 2024-03-24T10:35:25.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369002f070 con 0x7f3684004d00 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f368ffff640 1 -- 172.21.15.1:0/60883900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36900349f0 con 0x7f3684004d00 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 -- 172.21.15.1:0/60883900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3684004d00 msgr2=0x7f3684005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 --2- 172.21.15.1:0/60883900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3684004d00 0x7f3684005180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f369000a7d0 tx=0x7f36840057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369dc60640 1 -- 172.21.15.1:0/60883900 reap_dead start 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 -- 172.21.15.1:0/60883900 shutdown_connections 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 -- 172.21.15.1:0/60883900 >> 172.21.15.1:0/60883900 conn(0x7f36980d3ee0 msgr2=0x7f36980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 -- 172.21.15.1:0/60883900 shutdown_connections 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.637+0000 7f369f6c3640 1 -- 172.21.15.1:0/60883900 wait complete. 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.638+0000 7f369f6c3640 1 Processor -- start 2024-03-24T10:35:25.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.638+0000 7f369f6c3640 1 -- start start 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36980b45b0 0x7f36981577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36980b4990 0x7f36980b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3698160860 con 0x7f36980b7da0 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f369815ff70 con 0x7f36980b4990 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369f6c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f36980b8390 con 0x7f36980b45b0 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49302/0 (socket says 172.21.15.1:49302) 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 -- 172.21.15.1:0/2882692395 learned_addr learned my addr 172.21.15.1:0/2882692395 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369cc5e640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36980b4990 0x7f36980b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369d45f640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36980b45b0 0x7f36981577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 -- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36980b45b0 msgr2=0x7f36981577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36980b45b0 0x7f36981577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 -- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36980b4990 msgr2=0x7f36980b7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36980b4990 0x7f36980b7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.639+0000 7f369dc60640 1 -- 172.21.15.1:0/2882692395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3698160210 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f369dc60640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3690001c30 tx=0x7f3690001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690034840 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36900349a0 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36981604f0 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690034d10 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.640+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f369814b7f0 con 0x7f36980b7da0 2024-03-24T10:35:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.641+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3698160680 con 0x7f36980b7da0 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.641+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3690031030 con 0x7f36980b7da0 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.642+0000 7f368dffb640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 0x7f367c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.642+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f36900ac950 con 0x7f36980b7da0 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.642+0000 7f369d45f640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 0x7f367c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.642+0000 7f369d45f640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 0x7f367c06dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f36940099e0 tx=0x7f3694008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.646+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3690077270 con 0x7f36980b7da0 2024-03-24T10:35:25.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.828+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3698160680 con 0x7f36980b7da0 2024-03-24T10:35:25.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.829+0000 7f368dffb640 1 -- 172.21.15.1:0/2882692395 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f36900011c0 con 0x7f36980b7da0 2024-03-24T10:35:25.830 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 msgr2=0x7f367c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 0x7f367c06dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f36940099e0 tx=0x7f3694008040 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 msgr2=0x7f3698160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3690001c30 tx=0x7f3690001d00 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 shutdown_connections 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f367c06b810 0x7f367c06dcd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36980b7da0 0x7f3698160cf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36980b4990 0x7f36980b7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 --2- 172.21.15.1:0/2882692395 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f36980b45b0 0x7f36981577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 >> 172.21.15.1:0/2882692395 conn(0x7f36980d3ee0 msgr2=0x7f369806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 shutdown_connections 2024-03-24T10:35:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.831+0000 7f369f6c3640 1 -- 172.21.15.1:0/2882692395 wait complete. 2024-03-24T10:35:25.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:25.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.946+0000 7fbbf74af640 1 Processor -- start 2024-03-24T10:35:25.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.946+0000 7fbbf74af640 1 -- start start 2024-03-24T10:35:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbf00d2930 0x7fbbf00cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbbf00cd970 0x7fbbf00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf00d0e30 con 0x7fbbf00d2930 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf00cf8e0 con 0x7fbbf00cd970 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf74af640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf00cfb30 con 0x7fbbf015c1e0 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbf00d2930 0x7fbbf00cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbf00d2930 0x7fbbf00cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49304/0 (socket says 172.21.15.1:49304) 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 -- 172.21.15.1:0/3822705486 learned_addr learned my addr 172.21.15.1:0/3822705486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf5a4c640 1 --2- 172.21.15.1:0/3822705486 >> v2:172.21.15.3:3300/0 conn(0x7fbbf00cd970 0x7fbbf00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf524b640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 -- 172.21.15.1:0/3822705486 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 msgr2=0x7fbbf00d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 -- 172.21.15.1:0/3822705486 >> v2:172.21.15.3:3300/0 conn(0x7fbbf00cd970 msgr2=0x7fbbf00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:25.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/3822705486 >> v2:172.21.15.3:3300/0 conn(0x7fbbf00cd970 0x7fbbf00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf4a4a640 1 -- 172.21.15.1:0/3822705486 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf00cfe10 con 0x7fbbf00d2930 2024-03-24T10:35:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.947+0000 7fbbf524b640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.948+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00d2930 0x7fbbf00cd430 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00c54b0 tx=0x7fbbe002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae89d1fefa4ef0ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.948+0000 7fbbe7fff640 1 -- 172.21.15.1:0/3822705486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0030860 con 0x7fbbf00d2930 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.948+0000 7fbbe7fff640 1 -- 172.21.15.1:0/3822705486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe00309c0 con 0x7fbbf00d2930 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.948+0000 7fbbe7fff640 1 -- 172.21.15.1:0/3822705486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0030d50 con 0x7fbbf00d2930 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 -- 172.21.15.1:0/3822705486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00d2930 msgr2=0x7fbbf00cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00d2930 0x7fbbf00cd430 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00c54b0 tx=0x7fbbe002b9a0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 -- 172.21.15.1:0/3822705486 shutdown_connections 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 --2- 172.21.15.1:0/3822705486 >> v2:172.21.15.3:3300/0 conn(0x7fbbf00cd970 0x7fbbf00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00d2930 0x7fbbf00cd430 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 --2- 172.21.15.1:0/3822705486 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 -- 172.21.15.1:0/3822705486 >> 172.21.15.1:0/3822705486 conn(0x7fbbf00d1d10 msgr2=0x7fbbf00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 -- 172.21.15.1:0/3822705486 shutdown_connections 2024-03-24T10:35:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.949+0000 7fbbf74af640 1 -- 172.21.15.1:0/3822705486 wait complete. 2024-03-24T10:35:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.951+0000 7fbbf74af640 1 Processor -- start 2024-03-24T10:35:25.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.951+0000 7fbbf74af640 1 -- start start 2024-03-24T10:35:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 0x7fbbf00b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf009f690 con 0x7fbbf00cd970 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0152880 con 0x7fbbf00d2930 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf74af640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf009fb20 con 0x7fbbf015c1e0 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49312/0 (socket says 172.21.15.1:49312) 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 -- 172.21.15.1:0/153912844 learned_addr learned my addr 172.21.15.1:0/153912844 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 0x7fbbf00b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf5a4c640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 -- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 msgr2=0x7fbbf00b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 -- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 msgr2=0x7fbbf00b64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 0x7fbbf00b64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf524b640 1 -- 172.21.15.1:0/153912844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe00094d0 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.952+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 0x7fbbf00b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbf524b640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00d7d0 tx=0x7fbbec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec004030 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbec004190 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf009ea80 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec004520 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.953+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf009f040 con 0x7fbbf00cd970 2024-03-24T10:35:25.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.954+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf004f470 con 0x7fbbf00cd970 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.954+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbbec00b6c0 con 0x7fbbf00cd970 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.955+0000 7fbbe5ffb640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 0x7fbbd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.955+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fbbec08a570 con 0x7fbbf00cd970 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.955+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 0x7fbbd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.955+0000 7fbbf4a4a640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 0x7fbbd406dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0036000 tx=0x7fbbe0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:25.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:25.959+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbbec054e00 con 0x7fbbf00cd970 2024-03-24T10:35:26.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:35:26.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:35:26.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.211+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbbf00b7c90 con 0x7fbbf00cd970 2024-03-24T10:35:26.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.211+0000 7fbbe5ffb640 1 -- 172.21.15.1:0/153912844 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7fbbec0584c0 con 0x7fbbf00cd970 2024-03-24T10:35:26.213 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 msgr2=0x7fbbd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 0x7fbbd406dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0036000 tx=0x7fbbe0008030 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 msgr2=0x7fbbf0152340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00d7d0 tx=0x7fbbec00dc90 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 shutdown_connections 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbbd406b810 0x7fbbd406dcd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbbf015c1e0 0x7fbbf00b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbf00d2930 0x7fbbf00b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 --2- 172.21.15.1:0/153912844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbf00cd970 0x7fbbf0152340 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 >> 172.21.15.1:0/153912844 conn(0x7fbbf00d1d10 msgr2=0x7fbbf00cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 shutdown_connections 2024-03-24T10:35:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.213+0000 7fbbf74af640 1 -- 172.21.15.1:0/153912844 wait complete. 2024-03-24T10:35:26.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:26.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.332+0000 7f4d1c29d640 1 Processor -- start 2024-03-24T10:35:26.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.332+0000 7f4d1c29d640 1 -- start start 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d1415e390 0x7f4d140d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 0x7f4d140d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 0x7f4d140b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4d140b0b70 con 0x7f4d1415e390 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4d140d1d60 con 0x7f4d140b2ed0 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1c29d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4d140b0de0 con 0x7f4d140b1300 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d1415e390 0x7f4d140d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d1415e390 0x7f4d140d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49318/0 (socket says 172.21.15.1:49318) 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 -- 172.21.15.1:0/2071581453 learned_addr learned my addr 172.21.15.1:0/2071581453 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d19838640 1 --2- 172.21.15.1:0/2071581453 >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 0x7f4d140d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 -- 172.21.15.1:0/2071581453 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 msgr2=0x7f4d140b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a83a640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 0x7f4d140b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 0x7f4d140b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 -- 172.21.15.1:0/2071581453 >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 msgr2=0x7f4d140d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2071581453 >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 0x7f4d140d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a039640 1 -- 172.21.15.1:0/2071581453 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1415e770 con 0x7f4d1415e390 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d19838640 1 --2- 172.21.15.1:0/2071581453 >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 0x7f4d140d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.333+0000 7f4d1a83a640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 0x7f4d140b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d1415e390 0x7f4d140d2310 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4d1406b500 tx=0x7f4d1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1444efd722df4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d19037640 1 -- 172.21.15.1:0/2071581453 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10031870 con 0x7f4d1415e390 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d19037640 1 -- 172.21.15.1:0/2071581453 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d100319d0 con 0x7f4d1415e390 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d19037640 1 -- 172.21.15.1:0/2071581453 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10031d60 con 0x7f4d1415e390 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2071581453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d1415e390 msgr2=0x7f4d140d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d1415e390 0x7f4d140d2310 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4d1406b500 tx=0x7f4d1002b910 comp rx=0 tx=0).stop 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2071581453 shutdown_connections 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b1300 0x7f4d140b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2071581453 >> v2:172.21.15.3:3300/0 conn(0x7f4d140b2ed0 0x7f4d140d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2071581453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d1415e390 0x7f4d140d2310 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2071581453 >> 172.21.15.1:0/2071581453 conn(0x7f4d140d3ee0 msgr2=0x7f4d140d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2071581453 shutdown_connections 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.335+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2071581453 wait complete. 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 Processor -- start 2024-03-24T10:35:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 -- start start 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 0x7f4d1415e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 0x7f4d141502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1415ef30 con 0x7f4d140b1300 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1415f190 con 0x7f4d1415e390 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1c29d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1415f470 con 0x7f4d140b2ed0 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1a039640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1a039640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49322/0 (socket says 172.21.15.1:49322) 2024-03-24T10:35:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.337+0000 7f4d1a039640 1 -- 172.21.15.1:0/2057467206 learned_addr learned my addr 172.21.15.1:0/2057467206 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d19838640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 0x7f4d1415e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a83a640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 0x7f4d141502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 -- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 msgr2=0x7f4d1415e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 0x7f4d1415e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 -- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 msgr2=0x7f4d141502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 0x7f4d141502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 -- 172.21.15.1:0/2057467206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d100094d0 con 0x7f4d140b1300 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d19838640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 0x7f4d1415e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a83a640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 0x7f4d141502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1a039640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4d1002b950 tx=0x7f4d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10031860 con 0x7f4d140b1300 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d100443f0 con 0x7f4d140b1300 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1415f750 con 0x7f4d140b1300 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d100447a0 con 0x7f4d140b1300 2024-03-24T10:35:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.338+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1415fd10 con 0x7f4d140b1300 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.339+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d1415ff00 con 0x7f4d140b1300 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.340+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4d1004d450 con 0x7f4d140b1300 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.340+0000 7f4d0affd640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 0x7f4cf806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.340+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f4d1002d040 con 0x7f4d140b1300 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.340+0000 7f4d19838640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 0x7f4cf806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.341+0000 7f4d19838640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 0x7f4cf806dd00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4d140b50d0 tx=0x7f4d04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.344+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4d10077410 con 0x7f4d140b1300 2024-03-24T10:35:26.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.634+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4d1404f4e0 con 0x7f4d140b1300 2024-03-24T10:35:26.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.634+0000 7f4d0affd640 1 -- 172.21.15.1:0/2057467206 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7f4d10035020 con 0x7f4d140b1300 2024-03-24T10:35:26.637 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:26.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 msgr2=0x7f4cf806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 0x7f4cf806dd00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4d140b50d0 tx=0x7f4d04008040 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 msgr2=0x7f4d14160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4d1002b950 tx=0x7f4d100046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 shutdown_connections 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4cf806b840 0x7f4cf806dd00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d1415e390 0x7f4d141502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4d140b2ed0 0x7f4d1415e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 --2- 172.21.15.1:0/2057467206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d140b1300 0x7f4d14160230 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 >> 172.21.15.1:0/2057467206 conn(0x7f4d140d3ee0 msgr2=0x7f4d140d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 shutdown_connections 2024-03-24T10:35:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.637+0000 7f4d1c29d640 1 -- 172.21.15.1:0/2057467206 wait complete. 2024-03-24T10:35:26.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:26.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.752+0000 7f2f2fef4640 1 Processor -- start 2024-03-24T10:35:26.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.752+0000 7f2f2fef4640 1 -- start start 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2fef4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f28162610 0x7f2f280d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2fef4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f280c7a40 con 0x7f2f280b4e40 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2fef4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f280b55b0 con 0x7f2f281615f0 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2fef4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2f280cfd50 con 0x7f2f28162610 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2e491640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f28162610 0x7f2f280d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2e491640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f28162610 0x7f2f280d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49324/0 (socket says 172.21.15.1:49324) 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2e491640 1 -- 172.21.15.1:0/1926078551 learned_addr learned my addr 172.21.15.1:0/1926078551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2d48f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f280b4e40 0x7f2f280d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33374/0 (socket says 172.21.15.1:33374) 2024-03-24T10:35:26.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.753+0000 7f2f2e491640 1 -- 172.21.15.1:0/1926078551 >> v1:172.21.15.3:6789/0 conn(0x7f2f281615f0 legacy=0x7f2f28161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2e491640 1 -- 172.21.15.1:0/1926078551 >> v1:172.21.15.1:6789/0 conn(0x7f2f280b4e40 legacy=0x7f2f280d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2e491640 1 -- 172.21.15.1:0/1926078551 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f280b52d0 con 0x7f2f28162610 2024-03-24T10:35:26.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2cc8e640 1 -- 172.21.15.1:0/1926078551 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826239665 0 0) 0x7f2f280c7a40 con 0x7f2f280b4e40 2024-03-24T10:35:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2e491640 1 --2- 172.21.15.1:0/1926078551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28162610 0x7f2f280d4a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009e50 tx=0x7f2f2002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a787ebd9fafd740d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2cc8e640 1 -- 172.21.15.1:0/1926078551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2002cbb0 con 0x7f2f28162610 2024-03-24T10:35:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2cc8e640 1 -- 172.21.15.1:0/1926078551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2002cd10 con 0x7f2f28162610 2024-03-24T10:35:26.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.754+0000 7f2f2cc8e640 1 -- 172.21.15.1:0/1926078551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20034610 con 0x7f2f28162610 2024-03-24T10:35:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1926078551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28162610 msgr2=0x7f2f280d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1926078551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28162610 0x7f2f280d4a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009e50 tx=0x7f2f2002c010 comp rx=0 tx=0).stop 2024-03-24T10:35:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1926078551 shutdown_connections 2024-03-24T10:35:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1926078551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f28162610 0x7f2f280d4a10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1926078551 >> 172.21.15.1:0/1926078551 conn(0x7f2f280d3ef0 msgr2=0x7f2f280d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1926078551 shutdown_connections 2024-03-24T10:35:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.755+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1926078551 wait complete. 2024-03-24T10:35:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.756+0000 7f2f2fef4640 1 Processor -- start 2024-03-24T10:35:26.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.756+0000 7f2f2fef4640 1 -- start start 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 0x7f2f280cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f28162610 0x7f2f28155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2f280b6bb0 con 0x7f2f281f6f70 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f2f281f74c0 con 0x7f2f28162610 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2fef4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2f281f77a0 con 0x7f2f281615f0 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49328/0 (socket says 172.21.15.1:49328) 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 -- 172.21.15.1:0/1170903690 learned_addr learned my addr 172.21.15.1:0/1170903690 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2d48f640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f28162610 0x7f2f28155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2dc90640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 0x7f2f280cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 -- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 msgr2=0x7f2f280cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 0x7f2f280cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 -- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f28162610 msgr2=0x7f2f28155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f28162610 0x7f2f28155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2e491640 1 -- 172.21.15.1:0/1170903690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f20009b30 con 0x7f2f281f6f70 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.757+0000 7f2f2dc90640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 0x7f2f280cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f2e491640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f20036040 tx=0x7f2f20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2003e450 con 0x7f2f281f6f70 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2003e5d0 con 0x7f2f281f6f70 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f281f7a80 con 0x7f2f281f6f70 2024-03-24T10:35:26.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2003e980 con 0x7f2f281f6f70 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.758+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f280ce240 con 0x7f2f281f6f70 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.759+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f280b6d20 con 0x7f2f281f6f70 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.759+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f2003eae0 con 0x7f2f281f6f70 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.760+0000 7f2f1e7fc640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 0x7f2f0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.760+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f2f200ad160 con 0x7f2f281f6f70 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.760+0000 7f2f2dc90640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 0x7f2f0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.760+0000 7f2f2dc90640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 0x7f2f0c06dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f60 tx=0x7f2f24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:26.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.764+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f20077a80 con 0x7f2f281f6f70 2024-03-24T10:35:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.942+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f2f2804f470 con 0x7f2f281f6f70 2024-03-24T10:35:26.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.943+0000 7f2f1e7fc640 1 -- 172.21.15.1:0/1170903690 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f2f2007b140 con 0x7f2f281f6f70 2024-03-24T10:35:26.944 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:26.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 msgr2=0x7f2f0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 0x7f2f0c06dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2f24000f60 tx=0x7f2f24009040 comp rx=0 tx=0).stop 2024-03-24T10:35:26.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 msgr2=0x7f2f280d2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:26.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f20036040 tx=0x7f2f20004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:26.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 shutdown_connections 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f2f0c06b810 0x7f2f0c06dcd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f281f6f70 0x7f2f280d2660 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f28162610 0x7f2f28155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 --2- 172.21.15.1:0/1170903690 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f2f281615f0 0x7f2f280cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 >> 172.21.15.1:0/1170903690 conn(0x7f2f280d3ef0 msgr2=0x7f2f281619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 shutdown_connections 2024-03-24T10:35:26.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:26.945+0000 7f2f2fef4640 1 -- 172.21.15.1:0/1170903690 wait complete. 2024-03-24T10:35:26.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:35:27.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.058+0000 7f25c7ba7640 1 Processor -- start 2024-03-24T10:35:27.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.058+0000 7f25c7ba7640 1 -- start start 2024-03-24T10:35:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c7ba7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25c00d2bb0 0x7f25c00b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c7ba7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c015e770 con 0x7f25c015e390 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c7ba7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c00d4e40 con 0x7f25c00d1000 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c7ba7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f25c00b3190 con 0x7f25c00d2bb0 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5943640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f25c015e390 0x7f25c00d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33386/0 (socket says 172.21.15.1:33386) 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5943640 1 -- 172.21.15.1:0/1261614423 learned_addr learned my addr 172.21.15.1:0/1261614423 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5142640 1 --2- 172.21.15.1:0/1261614423 >> v2:172.21.15.1:3300/0 conn(0x7f25c00d2bb0 0x7f25c00b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5142640 1 -- 172.21.15.1:0/1261614423 >> v1:172.21.15.3:6789/0 conn(0x7f25c00d1000 legacy=0x7f25c00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5142640 1 -- 172.21.15.1:0/1261614423 >> v1:172.21.15.1:6789/0 conn(0x7f25c015e390 legacy=0x7f25c00d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c4941640 1 -- 172.21.15.1:0/1261614423 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242911084 0 0) 0x7f25bc001490 con 0x7f25c015e390 2024-03-24T10:35:27.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.059+0000 7f25c5142640 1 -- 172.21.15.1:0/1261614423 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c00d1e40 con 0x7f25c00d2bb0 2024-03-24T10:35:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.060+0000 7f25c5142640 1 --2- 172.21.15.1:0/1261614423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c00b3b90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0f30ef6073c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.060+0000 7f25c4941640 1 -- 172.21.15.1:0/1261614423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b002f050 con 0x7f25c00d2bb0 2024-03-24T10:35:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.060+0000 7f25c4941640 1 -- 172.21.15.1:0/1261614423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0031700 con 0x7f25c00d2bb0 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.060+0000 7f25c4941640 1 -- 172.21.15.1:0/1261614423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0031a90 con 0x7f25c00d2bb0 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 -- 172.21.15.1:0/1261614423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 msgr2=0x7f25c00b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/1261614423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c00b3b90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b002b7a0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 -- 172.21.15.1:0/1261614423 shutdown_connections 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/1261614423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c00b3b90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 -- 172.21.15.1:0/1261614423 >> 172.21.15.1:0/1261614423 conn(0x7f25c00d3ec0 msgr2=0x7f25c00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 -- 172.21.15.1:0/1261614423 shutdown_connections 2024-03-24T10:35:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.061+0000 7f25c7ba7640 1 -- 172.21.15.1:0/1261614423 wait complete. 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.062+0000 7f25c7ba7640 1 Processor -- start 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.062+0000 7f25c7ba7640 1 -- start start 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 0x7f25c00a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f25c015e390 0x7f25c009dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f25c01558b0 con 0x7f25c00d2bb0 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f25c009e150 con 0x7f25c00d1000 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c7ba7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f25c009ec90 con 0x7f25c015e390 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49354/0 (socket says 172.21.15.1:49354) 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 -- 172.21.15.1:0/2240220599 learned_addr learned my addr 172.21.15.1:0/2240220599 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5943640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 0x7f25c00a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 -- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f25c015e390 msgr2=0x7f25c009dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c6144640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f25c015e390 0x7f25c009dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f25c015e390 0x7f25c009dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 -- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 msgr2=0x7f25c00a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 0x7f25c00a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 -- 172.21.15.1:0/2240220599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b0009400 con 0x7f25c00d2bb0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5943640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 0x7f25c00a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.063+0000 7f25c5142640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.064+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0033c30 con 0x7f25c00d2bb0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.064+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0033d90 con 0x7f25c00d2bb0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.064+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c009e3f0 con 0x7f25c00d2bb0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.064+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b003d6b0 con 0x7f25c00d2bb0 2024-03-24T10:35:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.064+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c009e9b0 con 0x7f25c00d2bb0 2024-03-24T10:35:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.065+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c004f4e0 con 0x7f25c00d2bb0 2024-03-24T10:35:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.065+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25b003a040 con 0x7f25c00d2bb0 2024-03-24T10:35:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.066+0000 7f25b67fc640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.066+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f25b00ac230 con 0x7f25c00d2bb0 2024-03-24T10:35:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.066+0000 7f25c5943640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.066+0000 7f25c5943640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 0x7f25a406dd00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25bc007fd0 tx=0x7f25bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.070+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f25b0076b50 con 0x7f25c00d2bb0 2024-03-24T10:35:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.264+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f25c00d2280 con 0x7f25c00d2bb0 2024-03-24T10:35:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.264+0000 7f25b67fc640 1 -- 172.21.15.1:0/2240220599 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) ==== 80+0+0 (secure 0 0 0) 0x7f25b007a210 con 0x7f25c00d2bb0 2024-03-24T10:35:27.265 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:35:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 msgr2=0x7f25a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 0x7f25a406dd00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25bc007fd0 tx=0x7f25bc009040 comp rx=0 tx=0).stop 2024-03-24T10:35:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 msgr2=0x7f25c0155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b00046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 shutdown_connections 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f25c015e390 0x7f25c009dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c00d2bb0 0x7f25c0155370 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 --2- 172.21.15.1:0/2240220599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25c00d1000 0x7f25c00a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 >> 172.21.15.1:0/2240220599 conn(0x7f25c00d3ec0 msgr2=0x7f25c00cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.266+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 shutdown_connections 2024-03-24T10:35:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.267+0000 7f25c7ba7640 1 -- 172.21.15.1:0/2240220599 wait complete. 2024-03-24T10:35:27.279 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:35:27.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:27.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:35:27.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[54414]: fuse finished with error 0 and tester_r 0 2024-03-24T10:35:27.306 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:27.350 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:35:27.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:27.350 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:35:27.365 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-24T10:35:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 Processor -- start 2024-03-24T10:35:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 -- start start 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f29ec15a4d0 0x7f29ec06c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29ec06d970 0x7f29ec070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec15a8b0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec0cc270 con 0x7f29ec06d970 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.513+0000 7f29f2a62640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec0b4230 con 0x7f29ec15a4d0 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29ec06d970 0x7f29ec070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29ec06d970 0x7f29ec070cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49358/0 (socket says 172.21.15.1:49358) 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 -- 172.21.15.1:0/2307526028 learned_addr learned my addr 172.21.15.1:0/2307526028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29eb7fe640 1 --1- 172.21.15.1:0/2307526028 >> v1:172.21.15.1:6789/0 conn(0x7f29ec0b4ca0 0x7f29ec06d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33398/0 (socket says 172.21.15.1:33398) 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29ebfff640 1 --2- 172.21.15.1:0/2307526028 >> v2:172.21.15.3:3300/0 conn(0x7f29ec15a4d0 0x7f29ec06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 -- 172.21.15.1:0/2307526028 >> v2:172.21.15.3:3300/0 conn(0x7f29ec15a4d0 msgr2=0x7f29ec06c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 --2- 172.21.15.1:0/2307526028 >> v2:172.21.15.3:3300/0 conn(0x7f29ec15a4d0 0x7f29ec06c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 -- 172.21.15.1:0/2307526028 >> v1:172.21.15.1:6789/0 conn(0x7f29ec0b4ca0 legacy=0x7f29ec06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29eaffd640 1 -- 172.21.15.1:0/2307526028 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654914893 0 0) 0x7f29ec15a8b0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.514+0000 7f29f0fff640 1 -- 172.21.15.1:0/2307526028 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec0b4420 con 0x7f29ec06d970 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.515+0000 7f29f0fff640 1 --2- 172.21.15.1:0/2307526028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec06d970 0x7f29ec070cf0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0d0ac0 tx=0x7f29e402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47b52b386fb2cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.515+0000 7f29eaffd640 1 -- 172.21.15.1:0/2307526028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e400b0d0 con 0x7f29ec06d970 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.515+0000 7f29eaffd640 1 -- 172.21.15.1:0/2307526028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e400b230 con 0x7f29ec06d970 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.515+0000 7f29eaffd640 1 -- 172.21.15.1:0/2307526028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4009600 con 0x7f29ec06d970 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 -- 172.21.15.1:0/2307526028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec06d970 msgr2=0x7f29ec070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 --2- 172.21.15.1:0/2307526028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec06d970 0x7f29ec070cf0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0d0ac0 tx=0x7f29e402e170 comp rx=0 tx=0).stop 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 -- 172.21.15.1:0/2307526028 shutdown_connections 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 --2- 172.21.15.1:0/2307526028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec06d970 0x7f29ec070cf0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 --2- 172.21.15.1:0/2307526028 >> v2:172.21.15.3:3300/0 conn(0x7f29ec15a4d0 0x7f29ec06c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 -- 172.21.15.1:0/2307526028 >> 172.21.15.1:0/2307526028 conn(0x7f29ec06b4f0 msgr2=0x7f29ec06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 -- 172.21.15.1:0/2307526028 shutdown_connections 2024-03-24T10:35:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.516+0000 7f29f2a62640 1 -- 172.21.15.1:0/2307526028 wait complete. 2024-03-24T10:35:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.517+0000 7f29f2a62640 1 Processor -- start 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.517+0000 7f29f2a62640 1 -- start start 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29ec06d970 0x7f29ec15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29ec15a4d0 0x7f29ec146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec1f4bf0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec1f4120 con 0x7f29ec15a4d0 2024-03-24T10:35:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f2a62640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f29ec146ab0 con 0x7f29ec06d970 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49372/0 (socket says 172.21.15.1:49372) 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29eb7fe640 1 -- 172.21.15.1:0/3325822358 learned_addr learned my addr 172.21.15.1:0/3325822358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29ebfff640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29ec06d970 0x7f29ec15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.518+0000 7f29f0fff640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29ec15a4d0 0x7f29ec146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 -- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29ec06d970 msgr2=0x7f29ec15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29ec06d970 0x7f29ec15af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 -- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29ec15a4d0 msgr2=0x7f29ec146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29ec15a4d0 0x7f29ec146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 -- 172.21.15.1:0/3325822358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e400bd40 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29eb7fe640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f29e000c4d0 tx=0x7f29e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00074e0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e0007660 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0004a20 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1f7370 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.519+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1f4630 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.520+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec04f470 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.520+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f29e0004b80 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.521+0000 7f29e8ff9640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 0x7f29d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.521+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f29e00898e0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.521+0000 7f29ebfff640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 0x7f29d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:27.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.521+0000 7f29ebfff640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 0x7f29d806dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f29dc006fd0 tx=0x7f29dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.525+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29e0054170 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.779+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f29ec0b3310 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.781+0000 7f29e8ff9640 1 -- 172.21.15.1:0/3325822358 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) ==== 184+0+141 (secure 0 0 0) 0x7f29e00180e0 con 0x7f29ec0b4ca0 2024-03-24T10:35:27.782 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:35:27.782 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:35:27.783 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:35:27.783 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:35:27.783 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-24T10:35:27.783 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:35:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 msgr2=0x7f29d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 0x7f29d806dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f29dc006fd0 tx=0x7f29dc008040 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 msgr2=0x7f29ec1f3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f29e000c4d0 tx=0x7f29e000c990 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 shutdown_connections 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f29d806b810 0x7f29d806dcd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29ec15a4d0 0x7f29ec146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29ec0b4ca0 0x7f29ec1f3be0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 --2- 172.21.15.1:0/3325822358 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29ec06d970 0x7f29ec15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 >> 172.21.15.1:0/3325822358 conn(0x7f29ec06b4f0 msgr2=0x7f29ec0b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 shutdown_connections 2024-03-24T10:35:27.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:27.783+0000 7f29f2a62640 1 -- 172.21.15.1:0/3325822358 wait complete. 2024-03-24T10:35:27.796 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T10:35:27.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:35:27.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:35:27.886 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T10:35:27.886 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:35:27.886 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:27.886 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:27.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:35:27.901 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:35:27.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:27.901 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:35:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:27.958 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T10:35:27.958 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:35:27.959 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:35:27.959 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:27.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:35:27.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:35:27.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:27.984 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T10:35:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:28.042 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T10:35:28.043 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:35:28.043 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:28.043 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:28.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:35:28.067 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:35:28.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:28.067 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T10:35:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T10:35:28.125 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:35:28.125 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:35:28.125 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:28.143 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:35:28.144 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:35:28.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:35:28.144 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T10:35:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:35:28.202 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T10:35:28.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:28.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.302+0000 7f4beaa47640 1 Processor -- start 2024-03-24T10:35:28.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 -- start start 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4be415c000 0x7f4be40b4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 0x7f4be40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be40cd210 con 0x7f4be406f2a0 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4be40cfc40 con 0x7f4be415c000 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.303+0000 7f4beaa47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4be415c3e0 con 0x7f4be406ee20 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be3fff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4be415c000 0x7f4be40b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 0x7f4be40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be8fe4640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4be406f2a0 0x7f4be415fa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:36082/0 (socket says 172.21.15.1:36082) 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be8fe4640 1 -- 172.21.15.1:0/2443066764 learned_addr learned my addr 172.21.15.1:0/2443066764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be37fe640 1 -- 172.21.15.1:0/2443066764 >> v2:172.21.15.3:3300/0 conn(0x7f4be415c000 msgr2=0x7f4be40b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be37fe640 1 --2- 172.21.15.1:0/2443066764 >> v2:172.21.15.3:3300/0 conn(0x7f4be415c000 0x7f4be40b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be37fe640 1 -- 172.21.15.1:0/2443066764 >> v1:172.21.15.3:6789/0 conn(0x7f4be406f2a0 legacy=0x7f4be415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.304+0000 7f4be37fe640 1 -- 172.21.15.1:0/2443066764 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be40cf8d0 con 0x7f4be406ee20 2024-03-24T10:35:28.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.305+0000 7f4be37fe640 1 --2- 172.21.15.1:0/2443066764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 0x7f4be40d0060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3a7ade6bc7945d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4be2ffd640 1 -- 172.21.15.1:0/2443066764 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc02e070 con 0x7f4be406ee20 2024-03-24T10:35:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4be2ffd640 1 -- 172.21.15.1:0/2443066764 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc0315a0 con 0x7f4be406ee20 2024-03-24T10:35:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4be2ffd640 1 -- 172.21.15.1:0/2443066764 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc031930 con 0x7f4be406ee20 2024-03-24T10:35:28.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 -- 172.21.15.1:0/2443066764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 msgr2=0x7f4be40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 --2- 172.21.15.1:0/2443066764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 0x7f4be40d0060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).stop 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 -- 172.21.15.1:0/2443066764 shutdown_connections 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 --2- 172.21.15.1:0/2443066764 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be406ee20 0x7f4be40d0060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 --2- 172.21.15.1:0/2443066764 >> v2:172.21.15.3:3300/0 conn(0x7f4be415c000 0x7f4be40b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 -- 172.21.15.1:0/2443066764 >> 172.21.15.1:0/2443066764 conn(0x7f4be40b3770 msgr2=0x7f4be40b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 -- 172.21.15.1:0/2443066764 shutdown_connections 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.306+0000 7f4beaa47640 1 -- 172.21.15.1:0/2443066764 wait complete. 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.307+0000 7f4beaa47640 1 Processor -- start 2024-03-24T10:35:28.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.307+0000 7f4beaa47640 1 -- start start 2024-03-24T10:35:28.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 0x7f4be41568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 0x7f4be406fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4be4157490 con 0x7f4be406f2a0 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4be415d260 con 0x7f4be406ee20 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4beaa47640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4be415d540 con 0x7f4be415c000 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49384/0 (socket says 172.21.15.1:49384) 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4be37fe640 1 -- 172.21.15.1:0/3962078472 learned_addr learned my addr 172.21.15.1:0/3962078472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.308+0000 7f4be3fff640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 0x7f4be41568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be8fe4640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 0x7f4be406fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 -- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 msgr2=0x7f4be406fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 0x7f4be406fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 -- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 msgr2=0x7f4be41568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 0x7f4be41568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 -- 172.21.15.1:0/3962078472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc009400 con 0x7f4be406f2a0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be8fe4640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 0x7f4be406fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be3fff640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 0x7f4be41568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be37fe640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc034040 tx=0x7f4bcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc032840 con 0x7f4be406f2a0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc0329a0 con 0x7f4be406f2a0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be415d820 con 0x7f4be406f2a0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc032d50 con 0x7f4be406f2a0 2024-03-24T10:35:28.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.309+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be415dde0 con 0x7f4be406f2a0 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.310+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be404f4e0 con 0x7f4be406f2a0 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.311+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4bcc03b3f0 con 0x7f4be406f2a0 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.311+0000 7f4be0ff9640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 0x7f4bc406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.311+0000 7f4be3fff640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 0x7f4bc406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.311+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f4bcc0ac270 con 0x7f4be406f2a0 2024-03-24T10:35:28.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.312+0000 7f4be3fff640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 0x7f4bc406dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4be4070e60 tx=0x7f4bd400f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.316+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4bcc076b10 con 0x7f4be406f2a0 2024-03-24T10:35:28.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.534+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4be40b7be0 con 0x7f4be406f2a0 2024-03-24T10:35:28.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.535+0000 7f4be0ff9640 1 -- 172.21.15.1:0/3962078472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5165 (secure 0 0 0) 0x7f4bcc07a1d0 con 0x7f4be406f2a0 2024-03-24T10:35:28.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:28.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:35:02.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 msgr2=0x7f4bc406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 0x7f4bc406dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4be4070e60 tx=0x7f4bd400f040 comp rx=0 tx=0).stop 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 msgr2=0x7f4be406f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc034040 tx=0x7f4bcc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 shutdown_connections 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4bc406b810 0x7f4bc406dcd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4be415c000 0x7f4be406fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4be406f2a0 0x7f4be406f890 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 --2- 172.21.15.1:0/3962078472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4be406ee20 0x7f4be41568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 >> 172.21.15.1:0/3962078472 conn(0x7f4be40b3770 msgr2=0x7f4be40b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:28.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 shutdown_connections 2024-03-24T10:35:28.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.537+0000 7f4beaa47640 1 -- 172.21.15.1:0/3962078472 wait complete. 2024-03-24T10:35:28.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:28.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.652+0000 7f44b6e8f640 1 Processor -- start 2024-03-24T10:35:28.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.652+0000 7f44b6e8f640 1 -- start start 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b6e8f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f44b015bda0 0x7f44b015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b6e8f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44b00d00d0 0x7f44b0165a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b6e8f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b015d3c0 con 0x7f44b006c910 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b6e8f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f44b00b2db0 con 0x7f44b00d00d0 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b6e8f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f44b006e140 con 0x7f44b015bda0 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44b00d00d0 0x7f44b0165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44b00d00d0 0x7f44b0165a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49390/0 (socket says 172.21.15.1:49390) 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 -- 172.21.15.1:0/4186474217 learned_addr learned my addr 172.21.15.1:0/4186474217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b4c2b640 1 --2- 172.21.15.1:0/4186474217 >> v2:172.21.15.3:3300/0 conn(0x7f44b015bda0 0x7f44b015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 -- 172.21.15.1:0/4186474217 >> v2:172.21.15.3:3300/0 conn(0x7f44b015bda0 msgr2=0x7f44b015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 --2- 172.21.15.1:0/4186474217 >> v2:172.21.15.3:3300/0 conn(0x7f44b015bda0 0x7f44b015ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.653+0000 7f44b542c640 1 -- 172.21.15.1:0/4186474217 >> v1:172.21.15.3:6789/0 conn(0x7f44b006c910 legacy=0x7f44b00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.654+0000 7f44b542c640 1 -- 172.21.15.1:0/4186474217 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b006ce40 con 0x7f44b00d00d0 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.654+0000 7f44b542c640 1 --2- 172.21.15.1:0/4186474217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0165a20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f44b0075010 tx=0x7f44a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec245e08dec204f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.654+0000 7f44af7fe640 1 -- 172.21.15.1:0/4186474217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80317e0 con 0x7f44b00d00d0 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.654+0000 7f44af7fe640 1 -- 172.21.15.1:0/4186474217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a8031940 con 0x7f44b00d00d0 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.654+0000 7f44af7fe640 1 -- 172.21.15.1:0/4186474217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8031cd0 con 0x7f44b00d00d0 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 -- 172.21.15.1:0/4186474217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 msgr2=0x7f44b0165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/4186474217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0165a20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f44b0075010 tx=0x7f44a802b880 comp rx=0 tx=0).stop 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 -- 172.21.15.1:0/4186474217 shutdown_connections 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/4186474217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0165a20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/4186474217 >> v2:172.21.15.3:3300/0 conn(0x7f44b015bda0 0x7f44b015ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 -- 172.21.15.1:0/4186474217 >> 172.21.15.1:0/4186474217 conn(0x7f44b00b17a0 msgr2=0x7f44b00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 -- 172.21.15.1:0/4186474217 shutdown_connections 2024-03-24T10:35:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.655+0000 7f44b6e8f640 1 -- 172.21.15.1:0/4186474217 wait complete. 2024-03-24T10:35:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.657+0000 7f44b6e8f640 1 Processor -- start 2024-03-24T10:35:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.657+0000 7f44b6e8f640 1 -- start start 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44b006c910 0x7f44b00aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44b015bda0 0x7f44b0158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f44b014fb20 con 0x7f44b00d00d0 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f44b014fe00 con 0x7f44b015bda0 2024-03-24T10:35:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.658+0000 7f44b6e8f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f44b01500e0 con 0x7f44b006c910 2024-03-24T10:35:28.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49394/0 (socket says 172.21.15.1:49394) 2024-03-24T10:35:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 -- 172.21.15.1:0/3286123693 learned_addr learned my addr 172.21.15.1:0/3286123693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44b542c640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44b015bda0 0x7f44b0158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44b4c2b640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44b006c910 0x7f44b00aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 -- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44b006c910 msgr2=0x7f44b00aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44b006c910 0x7f44b00aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 -- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44b015bda0 msgr2=0x7f44b0158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44b015bda0 0x7f44b0158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 -- 172.21.15.1:0/3286123693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a80094d0 con 0x7f44b00d00d0 2024-03-24T10:35:28.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44affff640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c7c0 tx=0x7f44a000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a000aa50 con 0x7f44b00d00d0 2024-03-24T10:35:28.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a000abb0 con 0x7f44b00d00d0 2024-03-24T10:35:28.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b01503c0 con 0x7f44b00d00d0 2024-03-24T10:35:28.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0012420 con 0x7f44b00d00d0 2024-03-24T10:35:28.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.659+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b00af2a0 con 0x7f44b00d00d0 2024-03-24T10:35:28.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.661+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b01508e0 con 0x7f44b00d00d0 2024-03-24T10:35:28.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.661+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f44a0012580 con 0x7f44b00d00d0 2024-03-24T10:35:28.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.661+0000 7f44ad7fa640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 0x7f449006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.661+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f44a008a470 con 0x7f44b00d00d0 2024-03-24T10:35:28.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.661+0000 7f44b4c2b640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 0x7f449006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.662+0000 7f44b4c2b640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 0x7f449006dcd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f44980079a0 tx=0x7f4498008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.666+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44b01508e0 con 0x7f44b00d00d0 2024-03-24T10:35:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.849+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44b01508e0 con 0x7f44b00d00d0 2024-03-24T10:35:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.851+0000 7f44ad7fa640 1 -- 172.21.15.1:0/3286123693 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f44a00877c0 con 0x7f44b00d00d0 2024-03-24T10:35:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:28.852 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.852+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 msgr2=0x7f449006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.852+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 0x7f449006dcd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f44980079a0 tx=0x7f4498008040 comp rx=0 tx=0).stop 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.852+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 msgr2=0x7f44b0150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.852+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c7c0 tx=0x7f44a000cc80 comp rx=0 tx=0).stop 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 shutdown_connections 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f449006b810 0x7f449006dcd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44b015bda0 0x7f44b0158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b00d00d0 0x7f44b0150bc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 --2- 172.21.15.1:0/3286123693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f44b006c910 0x7f44b00aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 >> 172.21.15.1:0/3286123693 conn(0x7f44b00b17a0 msgr2=0x7f44b00b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 shutdown_connections 2024-03-24T10:35:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.853+0000 7f44b6e8f640 1 -- 172.21.15.1:0/3286123693 wait complete. 2024-03-24T10:35:28.865 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:34:51.593102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:34:52.680355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-24T10:35:02.227765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:28.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.972+0000 7f0dc3284640 1 Processor -- start 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.972+0000 7f0dc3284640 1 -- start start 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc3284640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0dbc0d2a20 0x7f0dbc0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc3284640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc0b0b70 con 0x7f0dbc0b2310 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc3284640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dbc0d1d60 con 0x7f0dbc15e390 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc3284640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0dbc0b0de0 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc081f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0dbc0d2a20 0x7f0dbc0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc1821640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0dbc0b2310 0x7f0dbc0cf010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33412/0 (socket says 172.21.15.1:33412) 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.973+0000 7f0dc1821640 1 -- 172.21.15.1:0/84908623 learned_addr learned my addr 172.21.15.1:0/84908623 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.974+0000 7f0dc081f640 1 -- 172.21.15.1:0/84908623 >> v1:172.21.15.3:6789/0 conn(0x7f0dbc15e390 legacy=0x7f0dbc0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.974+0000 7f0daffff640 1 -- 172.21.15.1:0/84908623 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804678088 0 0) 0x7f0dbc0b0b70 con 0x7f0dbc0b2310 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.974+0000 7f0dc081f640 1 -- 172.21.15.1:0/84908623 >> v1:172.21.15.1:6789/0 conn(0x7f0dbc0b2310 legacy=0x7f0dbc0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.974+0000 7f0dc081f640 1 -- 172.21.15.1:0/84908623 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc15e770 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.974+0000 7f0dc081f640 1 --2- 172.21.15.1:0/84908623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc0cead0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64160f00380f04e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0daffff640 1 -- 172.21.15.1:0/84908623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0031830 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0daffff640 1 -- 172.21.15.1:0/84908623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db0031990 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0daffff640 1 -- 172.21.15.1:0/84908623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0031d20 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 -- 172.21.15.1:0/84908623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0d2a20 msgr2=0x7f0dbc0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 --2- 172.21.15.1:0/84908623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc0cead0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 -- 172.21.15.1:0/84908623 shutdown_connections 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 --2- 172.21.15.1:0/84908623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc0cead0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 -- 172.21.15.1:0/84908623 >> 172.21.15.1:0/84908623 conn(0x7f0dbc0d3ee0 msgr2=0x7f0dbc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 -- 172.21.15.1:0/84908623 shutdown_connections 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.975+0000 7f0dc3284640 1 -- 172.21.15.1:0/84908623 wait complete. 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.976+0000 7f0dc3284640 1 Processor -- start 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.976+0000 7f0dc3284640 1 -- start start 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc14d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dbc15e390 0x7f0dbc0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc0cd8d0 con 0x7f0dbc0b2310 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc15d7b0 con 0x7f0dbc0d2a20 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc3284640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0dbc0ccc50 con 0x7f0dbc15e390 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc1020640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc1020640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49404/0 (socket says 172.21.15.1:49404) 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc1020640 1 -- 172.21.15.1:0/1323241676 learned_addr learned my addr 172.21.15.1:0/1323241676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:28.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.977+0000 7f0dc081f640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc14d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 -- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dbc15e390 msgr2=0x7f0dbc0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1821640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dbc15e390 0x7f0dbc0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dbc15e390 0x7f0dbc0cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 -- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 msgr2=0x7f0dbc14d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc14d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 -- 172.21.15.1:0/1323241676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db0009400 con 0x7f0dbc0b2310 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc081f640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc14d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc1020640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0db800b3f0 tx=0x7f0db800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8011830 con 0x7f0dbc0b2310 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db8011990 con 0x7f0dbc0b2310 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dbc0ccf30 con 0x7f0dbc0b2310 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8011d40 con 0x7f0dbc0b2310 2024-03-24T10:35:28.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.978+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dbc0cd4f0 con 0x7f0dbc0b2310 2024-03-24T10:35:28.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.979+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dbc0cd680 con 0x7f0dbc0b2310 2024-03-24T10:35:28.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.980+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0db801a400 con 0x7f0dbc0b2310 2024-03-24T10:35:28.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.980+0000 7f0dadffb640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 0x7f0da006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:28.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.980+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f0db808a180 con 0x7f0dbc0b2310 2024-03-24T10:35:28.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.980+0000 7f0dc081f640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 0x7f0da006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:28.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.980+0000 7f0dc081f640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 0x7f0da006dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:28.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:28.984+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0db8054a10 con 0x7f0dbc0b2310 2024-03-24T10:35:29.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.219+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0dbc0cdb40 con 0x7f0dbc0b2310 2024-03-24T10:35:29.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.219+0000 7f0dadffb640 1 -- 172.21.15.1:0/1323241676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f0db80580d0 con 0x7f0dbc0b2310 2024-03-24T10:35:29.220 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:29.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 msgr2=0x7f0da006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 0x7f0da006dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).stop 2024-03-24T10:35:29.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 msgr2=0x7f0dbc0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0db800b3f0 tx=0x7f0db800b8b0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 shutdown_connections 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0da006b810 0x7f0da006dcd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0dbc15e390 0x7f0dbc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0dbc0d2a20 0x7f0dbc14d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 --2- 172.21.15.1:0/1323241676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0dbc0b2310 0x7f0dbc0cc600 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 >> 172.21.15.1:0/1323241676 conn(0x7f0dbc0d3ee0 msgr2=0x7f0dbc15f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 shutdown_connections 2024-03-24T10:35:29.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.221+0000 7f0dc3284640 1 -- 172.21.15.1:0/1323241676 wait complete. 2024-03-24T10:35:29.235 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:35:29.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.342+0000 7f7d1144e640 1 Processor -- start 2024-03-24T10:35:29.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.342+0000 7f7d1144e640 1 -- start start 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d1144e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b4e40 0x7f7d0c0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d1144e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7d0c162610 0x7f7d0c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d1144e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0c0c7a40 con 0x7f7d0c1615f0 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d1144e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f7d0c0cfd50 con 0x7f7d0c162610 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d1144e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b55b0 con 0x7f7d0c0b4e40 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d0affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d0c1615f0 0x7f7d0c161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33416/0 (socket says 172.21.15.1:33416) 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d0affd640 1 -- 172.21.15.1:0/3648285876 learned_addr learned my addr 172.21.15.1:0/3648285876 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3648285876 >> v2:172.21.15.3:3300/0 conn(0x7f7d0c162610 0x7f7d0c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773122186 0 0) 0x7f7d0c0c7a40 con 0x7f7d0c1615f0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d03fff640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b4e40 0x7f7d0c0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.343+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cf8003520 con 0x7f7d0c1615f0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 172743947 0 0) 0x7f7cf8003520 con 0x7f7d0c1615f0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b4e40 msgr2=0x7f7d0c0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b4e40 0x7f7d0c0d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 >> v2:172.21.15.3:3300/0 conn(0x7f7d0c162610 msgr2=0x7f7d0c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 --2- 172.21.15.1:0/3648285876 >> v2:172.21.15.3:3300/0 conn(0x7f7d0c162610 0x7f7d0c0d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c0b52d0 con 0x7f7d0c1615f0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f7cfc001670 con 0x7f7d0c1615f0 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 >> v1:172.21.15.1:6789/0 conn(0x7f7d0c1615f0 legacy=0x7f7d0c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7cf8003970 0x7f7cf8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7cf80042d0 0x7f7cf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cf8004e10 0x7f7cf8005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0cfd50 con 0x7f7cf8004e10 2024-03-24T10:35:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b55b0 con 0x7f7cf80042d0 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf800bd00 con 0x7f7cf8003970 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cf8004e10 0x7f7cf8005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0affd640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7cf8003970 0x7f7cf8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d03fff640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7cf80042d0 0x7f7cf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7cf8003970 msgr2=0x7f7cf8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7cf8003970 0x7f7cf8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7cf80042d0 msgr2=0x7f7cf80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7cf80042d0 0x7f7cf80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.344+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3648285876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf8003520 con 0x7f7cf8004e10 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.345+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cf8004e10 0x7f7cf8005290 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009fc0 tx=0x7f7cf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768669088a9b77c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.345+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0402cc60 con 0x7f7cf8004e10 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.345+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0400f040 con 0x7f7cf8004e10 2024-03-24T10:35:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.345+0000 7f7d0a7fc640 1 -- 172.21.15.1:0/3648285876 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04034440 con 0x7f7cf8004e10 2024-03-24T10:35:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 -- 172.21.15.1:0/3648285876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cf8004e10 msgr2=0x7f7cf8005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3648285876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cf8004e10 0x7f7cf8005290 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009fc0 tx=0x7f7cf80057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3648285876 reap_dead start 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 -- 172.21.15.1:0/3648285876 shutdown_connections 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 -- 172.21.15.1:0/3648285876 >> 172.21.15.1:0/3648285876 conn(0x7f7d0c0d3ef0 msgr2=0x7f7d0c0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 -- 172.21.15.1:0/3648285876 shutdown_connections 2024-03-24T10:35:29.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.346+0000 7f7d1144e640 1 -- 172.21.15.1:0/3648285876 wait complete. 2024-03-24T10:35:29.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.348+0000 7f7d1144e640 1 Processor -- start 2024-03-24T10:35:29.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.348+0000 7f7d1144e640 1 -- start start 2024-03-24T10:35:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d0c15b590 0x7f7d0c0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b22b0 0x7f7d0c1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c06d7a0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b2820 con 0x7f7d0c15b590 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d1144e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b2b00 con 0x7f7d0c0b22b0 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49408/0 (socket says 172.21.15.1:49408) 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3661210274 learned_addr learned my addr 172.21.15.1:0/3661210274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0affd640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d0c15b590 0x7f7d0c0b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d03fff640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b22b0 0x7f7d0c1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b22b0 msgr2=0x7f7d0c1f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b22b0 0x7f7d0c1f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d0c15b590 msgr2=0x7f7d0c0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d0c15b590 0x7f7d0c0b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.349+0000 7f7d0b7fe640 1 -- 172.21.15.1:0/3661210274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c1f3960 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d0b7fe640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7d04037040 tx=0x7f7d04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04043070 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d04034bd0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c1f3c40 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04003990 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.350+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c0b4970 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.351+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0c1f37c0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.351+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7d0403e440 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.352+0000 7f7d037fe640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 0x7f7cec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.352+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f7d040abb30 con 0x7f7d0c06cde0 2024-03-24T10:35:29.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.352+0000 7f7d0affd640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 0x7f7cec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.352+0000 7f7d0affd640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 0x7f7cec06dd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc004840 tx=0x7f7cfc0078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.356+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d0c1f37c0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.536+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7d0c1f37c0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.536+0000 7f7d037fe640 1 -- 172.21.15.1:0/3661210274 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) ==== 104+0+11 (secure 0 0 0) 0x7f7d0c1f37c0 con 0x7f7d0c06cde0 2024-03-24T10:35:29.538 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 msgr2=0x7f7cec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 0x7f7cec06dd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc004840 tx=0x7f7cfc0078f0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 msgr2=0x7f7d0c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7d04037040 tx=0x7f7d04009750 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 shutdown_connections 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7cec06b840 0x7f7cec06dd00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0c06cde0 0x7f7d0c06d260 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d0c0b22b0 0x7f7d0c1f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 --2- 172.21.15.1:0/3661210274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d0c15b590 0x7f7d0c0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 >> 172.21.15.1:0/3661210274 conn(0x7f7d0c0d3ef0 msgr2=0x7f7d0c15e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 shutdown_connections 2024-03-24T10:35:29.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.538+0000 7f7d1144e640 1 -- 172.21.15.1:0/3661210274 wait complete. 2024-03-24T10:35:29.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:35:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 Processor -- start 2024-03-24T10:35:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 -- start start 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 0x7fc8a00d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc8a00b2ed0 0x7fc8a00d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 0x7fc8a00b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a00b0b70 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a00d1d60 con 0x7fc8a015e390 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.654+0000 7fc8a650f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a00b0de0 con 0x7fc8a00b1300 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc8a00b2ed0 0x7fc8a00d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc8a00b2ed0 0x7fc8a00d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49420/0 (socket says 172.21.15.1:49420) 2024-03-24T10:35:29.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 -- 172.21.15.1:0/2155474766 learned_addr learned my addr 172.21.15.1:0/2155474766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89ffff640 1 --2- 172.21.15.1:0/2155474766 >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 0x7fc8a00d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc8a4aac640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 0x7fc8a00b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 -- 172.21.15.1:0/2155474766 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 msgr2=0x7fc8a00b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 0x7fc8a00b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 -- 172.21.15.1:0/2155474766 >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 msgr2=0x7fc8a00d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 --2- 172.21.15.1:0/2155474766 >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 0x7fc8a00d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89f7fe640 1 -- 172.21.15.1:0/2155474766 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a015e770 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc89ffff640 1 --2- 172.21.15.1:0/2155474766 >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 0x7fc8a00d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.655+0000 7fc8a4aac640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 0x7fc8a00b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc89f7fe640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a00b2ed0 0x7fc8a00d2850 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc8a008c020 tx=0x7fc89402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9176485413364508 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc89effd640 1 -- 172.21.15.1:0/2155474766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc894031870 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc89effd640 1 -- 172.21.15.1:0/2155474766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8940319d0 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc89effd640 1 -- 172.21.15.1:0/2155474766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc894031d60 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 -- 172.21.15.1:0/2155474766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a00b2ed0 msgr2=0x7fc8a00d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a00b2ed0 0x7fc8a00d2850 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc8a008c020 tx=0x7fc89402b910 comp rx=0 tx=0).stop 2024-03-24T10:35:29.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 -- 172.21.15.1:0/2155474766 shutdown_connections 2024-03-24T10:35:29.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b1300 0x7fc8a00b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 --2- 172.21.15.1:0/2155474766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a00b2ed0 0x7fc8a00d2850 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 --2- 172.21.15.1:0/2155474766 >> v2:172.21.15.3:3300/0 conn(0x7fc8a015e390 0x7fc8a00d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.656+0000 7fc8a650f640 1 -- 172.21.15.1:0/2155474766 >> 172.21.15.1:0/2155474766 conn(0x7fc8a00d3ee0 msgr2=0x7fc8a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.657+0000 7fc8a650f640 1 -- 172.21.15.1:0/2155474766 shutdown_connections 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.657+0000 7fc8a650f640 1 -- 172.21.15.1:0/2155474766 wait complete. 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.658+0000 7fc8a650f640 1 Processor -- start 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.658+0000 7fc8a650f640 1 -- start start 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 0x7fc8a01612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b2ed0 0x7fc8a00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0160220 con 0x7fc8a015e390 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0160500 con 0x7fc8a00b1300 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a01607e0 con 0x7fc8a00b2ed0 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49428/0 (socket says 172.21.15.1:49428) 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 -- 172.21.15.1:0/3725959954 learned_addr learned my addr 172.21.15.1:0/3725959954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89ffff640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 0x7fc8a01612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89f7fe640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b2ed0 0x7fc8a00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 -- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b2ed0 msgr2=0x7fc8a00ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b2ed0 0x7fc8a00ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 -- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 msgr2=0x7fc8a01612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 0x7fc8a01612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 -- 172.21.15.1:0/3725959954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8940094d0 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89ffff640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 0x7fc8a01612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a4aac640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc898007e10 tx=0x7fc89800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898003b90 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89800f040 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a0160ac0 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89800de80 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.659+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a0158820 con 0x7fc8a015e390 2024-03-24T10:35:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.660+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a0160fe0 con 0x7fc8a015e390 2024-03-24T10:35:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.661+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc898004030 con 0x7fc8a015e390 2024-03-24T10:35:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.662+0000 7fc89cff9640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 0x7fc88c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.662+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fc89808a650 con 0x7fc8a015e390 2024-03-24T10:35:29.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.662+0000 7fc89ffff640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 0x7fc88c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.662+0000 7fc89ffff640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 0x7fc88c06dcd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc890006fd0 tx=0x7fc890008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.670+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8a0160fe0 con 0x7fc8a015e390 2024-03-24T10:35:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.851+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8a0160fe0 con 0x7fc8a015e390 2024-03-24T10:35:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.852+0000 7fc89cff9640 1 -- 172.21.15.1:0/3725959954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) ==== 100+0+11 (secure 0 0 0) 0x7fc8a0160fe0 con 0x7fc8a015e390 2024-03-24T10:35:29.853 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:35:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.853+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 msgr2=0x7fc88c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.853+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 0x7fc88c06dcd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc890006fd0 tx=0x7fc890008040 comp rx=0 tx=0).stop 2024-03-24T10:35:29.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 msgr2=0x7fc8a00cea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc898007e10 tx=0x7fc89800d4a0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 shutdown_connections 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc88c06b810 0x7fc88c06dcd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8a015e390 0x7fc8a00cea90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc8a00b2ed0 0x7fc8a00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 --2- 172.21.15.1:0/3725959954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8a00b1300 0x7fc8a01612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 >> 172.21.15.1:0/3725959954 conn(0x7fc8a00d3ee0 msgr2=0x7fc8a006cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 shutdown_connections 2024-03-24T10:35:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.854+0000 7fc8a650f640 1 -- 172.21.15.1:0/3725959954 wait complete. 2024-03-24T10:35:29.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-24T10:35:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.969+0000 7f01e65af640 1 Processor -- start 2024-03-24T10:35:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.969+0000 7f01e65af640 1 -- start start 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e65af640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01e00b3310 0x7f01e00cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e65af640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e0072620 con 0x7f01e015e320 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e65af640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e00d02b0 con 0x7f01e006b7b0 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e65af640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f01e015e700 con 0x7f01e00b3310 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e4b4c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01e00b3310 0x7f01e00cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e4b4c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01e00b3310 0x7f01e00cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49432/0 (socket says 172.21.15.1:49432) 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01e4b4c640 1 -- 172.21.15.1:0/2343403168 learned_addr learned my addr 172.21.15.1:0/2343403168 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.970+0000 7f01dffff640 1 --1- 172.21.15.1:0/2343403168 >> v1:172.21.15.1:6789/0 conn(0x7f01e015e320 0x7f01e006b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33420/0 (socket says 172.21.15.1:33420) 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01e4b4c640 1 -- 172.21.15.1:0/2343403168 >> v1:172.21.15.3:6789/0 conn(0x7f01e006b7b0 legacy=0x7f01e00ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:29.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01e4b4c640 1 -- 172.21.15.1:0/2343403168 >> v1:172.21.15.1:6789/0 conn(0x7f01e015e320 legacy=0x7f01e006b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01e4b4c640 1 -- 172.21.15.1:0/2343403168 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e00cfcf0 con 0x7f01e00b3310 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01e4b4c640 1 --2- 172.21.15.1:0/2343403168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e00b3310 0x7f01e00cf660 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e50 tx=0x7f01d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250c764d96248434 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01deffd640 1 -- 172.21.15.1:0/2343403168 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d802ca80 con 0x7f01e00b3310 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01deffd640 1 -- 172.21.15.1:0/2343403168 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d802cbe0 con 0x7f01e00b3310 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.971+0000 7f01deffd640 1 -- 172.21.15.1:0/2343403168 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80343f0 con 0x7f01e00b3310 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 -- 172.21.15.1:0/2343403168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e00b3310 msgr2=0x7f01e00cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 --2- 172.21.15.1:0/2343403168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e00b3310 0x7f01e00cf660 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e50 tx=0x7f01d802bee0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 -- 172.21.15.1:0/2343403168 shutdown_connections 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 --2- 172.21.15.1:0/2343403168 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e00b3310 0x7f01e00cf660 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 -- 172.21.15.1:0/2343403168 >> 172.21.15.1:0/2343403168 conn(0x7f01e00b2820 msgr2=0x7f01e00b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 -- 172.21.15.1:0/2343403168 shutdown_connections 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.972+0000 7f01e65af640 1 -- 172.21.15.1:0/2343403168 wait complete. 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.974+0000 7f01e65af640 1 Processor -- start 2024-03-24T10:35:29.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.974+0000 7f01e65af640 1 -- start start 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01e006b7b0 0x7f01e00b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f01e00b3310 0x7f01e01f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f01e015ecb0 con 0x7f01e015e320 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f01e015ef90 con 0x7f01e006b7b0 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e65af640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f01e015f270 con 0x7f01e00b3310 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e4b4c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e4b4c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49448/0 (socket says 172.21.15.1:49448) 2024-03-24T10:35:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.975+0000 7f01e4b4c640 1 -- 172.21.15.1:0/3173643763 learned_addr learned my addr 172.21.15.1:0/3173643763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01dffff640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01e006b7b0 0x7f01e00b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01df7fe640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f01e00b3310 0x7f01e01f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 -- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f01e00b3310 msgr2=0x7f01e01f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f01e00b3310 0x7f01e01f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 -- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01e006b7b0 msgr2=0x7f01e00b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01e006b7b0 0x7f01e00b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 -- 172.21.15.1:0/3173643763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d8009b30 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e4b4c640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f01d8036040 tx=0x7f01d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d803e450 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d803e5d0 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e015f550 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d803e980 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.976+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e00b5110 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.977+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e0069650 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.978+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f01d802cd50 con 0x7f01e015e320 2024-03-24T10:35:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.978+0000 7f01dcff9640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 0x7f01c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.978+0000 7f01dffff640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 0x7f01c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.978+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f01d80ace00 con 0x7f01e015e320 2024-03-24T10:35:29.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.979+0000 7f01dffff640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 0x7f01c006dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007fd0 tx=0x7f01d0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:29.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:29.982+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f01d8077860 con 0x7f01e015e320 2024-03-24T10:35:30.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.198+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f01e004f470 con 0x7f01e015e320 2024-03-24T10:35:30.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.198+0000 7f01dcff9640 1 -- 172.21.15.1:0/3173643763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) ==== 97+0+11 (secure 0 0 0) 0x7f01d8043020 con 0x7f01e015e320 2024-03-24T10:35:30.200 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 msgr2=0x7f01c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 0x7f01c006dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f01d0007fd0 tx=0x7f01d0007e80 comp rx=0 tx=0).stop 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 msgr2=0x7f01e01f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f01d8036040 tx=0x7f01d8004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 shutdown_connections 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f01c006b810 0x7f01c006dcd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01e015e320 0x7f01e01f7c60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f01e00b3310 0x7f01e01f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 --2- 172.21.15.1:0/3173643763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01e006b7b0 0x7f01e00b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 >> 172.21.15.1:0/3173643763 conn(0x7f01e00b2820 msgr2=0x7f01e00b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 shutdown_connections 2024-03-24T10:35:30.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.201+0000 7f01e65af640 1 -- 172.21.15.1:0/3173643763 wait complete. 2024-03-24T10:35:30.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.314+0000 7f4c2b84e640 1 Processor -- start 2024-03-24T10:35:30.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.314+0000 7f4c2b84e640 1 -- start start 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c2b84e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c240d1000 0x7f4c240b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c2b84e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2415e770 con 0x7f4c240d2bb0 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c2b84e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c240d4e40 con 0x7f4c2415e390 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c2b84e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4c240b3190 con 0x7f4c240d1000 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c29deb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c240d1000 0x7f4c240b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c28de9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4c240d2bb0 0x7f4c240b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33434/0 (socket says 172.21.15.1:33434) 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c29deb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c240d1000 0x7f4c240b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49454/0 (socket says 172.21.15.1:49454) 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.315+0000 7f4c28de9640 1 -- 172.21.15.1:0/2506159898 learned_addr learned my addr 172.21.15.1:0/2506159898 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.316+0000 7f4c29deb640 1 -- 172.21.15.1:0/2506159898 >> v1:172.21.15.3:6789/0 conn(0x7f4c2415e390 legacy=0x7f4c240d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.316+0000 7f4c29deb640 1 -- 172.21.15.1:0/2506159898 >> v1:172.21.15.1:6789/0 conn(0x7f4c240d2bb0 legacy=0x7f4c240b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.316+0000 7f4c29deb640 1 -- 172.21.15.1:0/2506159898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c240d1e40 con 0x7f4c240d1000 2024-03-24T10:35:30.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.316+0000 7f4c29deb640 1 --2- 172.21.15.1:0/2506159898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c240b42a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b713defe96dd7a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.316+0000 7f4c23fff640 1 -- 172.21.15.1:0/2506159898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c02f050 con 0x7f4c240d1000 2024-03-24T10:35:30.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c23fff640 1 -- 172.21.15.1:0/2506159898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c031700 con 0x7f4c240d1000 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c23fff640 1 -- 172.21.15.1:0/2506159898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c031a90 con 0x7f4c240d1000 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 -- 172.21.15.1:0/2506159898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 msgr2=0x7f4c240b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/2506159898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c240b42a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b7a0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 -- 172.21.15.1:0/2506159898 shutdown_connections 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/2506159898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c240b42a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 -- 172.21.15.1:0/2506159898 >> 172.21.15.1:0/2506159898 conn(0x7f4c240d3ec0 msgr2=0x7f4c240d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 -- 172.21.15.1:0/2506159898 shutdown_connections 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.317+0000 7f4c2b84e640 1 -- 172.21.15.1:0/2506159898 wait complete. 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.318+0000 7f4c2b84e640 1 Processor -- start 2024-03-24T10:35:30.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.318+0000 7f4c2b84e640 1 -- start start 2024-03-24T10:35:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c240d2bb0 0x7f4c240b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c2415e390 0x7f4c240b94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2415c030 con 0x7f4c240d1000 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4c241f4110 con 0x7f4c2415e390 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c2b84e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4c241f43f0 con 0x7f4c240d2bb0 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c295ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c295ea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49464/0 (socket says 172.21.15.1:49464) 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c295ea640 1 -- 172.21.15.1:0/1335921876 learned_addr learned my addr 172.21.15.1:0/1335921876 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.319+0000 7f4c28de9640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c240d2bb0 0x7f4c240b8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c29deb640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c2415e390 0x7f4c240b94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 -- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c240d2bb0 msgr2=0x7f4c240b8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c240d2bb0 0x7f4c240b8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 -- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c2415e390 msgr2=0x7f4c240b94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c2415e390 0x7f4c240b94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 -- 172.21.15.1:0/1335921876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c009400 con 0x7f4c240d1000 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c295ea640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400ba50 tx=0x7f4c1400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400cbe0 con 0x7f4c240d1000 2024-03-24T10:35:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1400cd80 con 0x7f4c240d1000 2024-03-24T10:35:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14014680 con 0x7f4c240d1000 2024-03-24T10:35:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c241f46d0 con 0x7f4c240d1000 2024-03-24T10:35:30.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.320+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c241f4c90 con 0x7f4c240d1000 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.321+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c2404f4e0 con 0x7f4c240d1000 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.321+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4c140147e0 con 0x7f4c240d1000 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.322+0000 7f4c21ffb640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 0x7f4c1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.322+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f4c1408a1b0 con 0x7f4c240d1000 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.322+0000 7f4c28de9640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 0x7f4c1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.322+0000 7f4c28de9640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 0x7f4c1006dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4c240b2a20 tx=0x7f4c18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.327+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4c14054b80 con 0x7f4c240d1000 2024-03-24T10:35:30.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.544+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c240d2280 con 0x7f4c240d1000 2024-03-24T10:35:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.544+0000 7f4c21ffb640 1 -- 172.21.15.1:0/1335921876 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5165 (secure 0 0 0) 0x7f4c14058240 con 0x7f4c240d1000 2024-03-24T10:35:30.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:30.546 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:35:02.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:35:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 msgr2=0x7f4c1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 0x7f4c1006dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4c240b2a20 tx=0x7f4c18009040 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 msgr2=0x7f4c2415baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400ba50 tx=0x7f4c1400bf10 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 shutdown_connections 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4c1006b810 0x7f4c1006dcd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c2415e390 0x7f4c240b94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c240d2bb0 0x7f4c240b8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 --2- 172.21.15.1:0/1335921876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c240d1000 0x7f4c2415baf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.546+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 >> 172.21.15.1:0/1335921876 conn(0x7f4c240d3ec0 msgr2=0x7f4c240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.547+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 shutdown_connections 2024-03-24T10:35:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.547+0000 7f4c2b84e640 1 -- 172.21.15.1:0/1335921876 wait complete. 2024-03-24T10:35:30.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:30.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.660+0000 7fd9ecec9640 1 Processor -- start 2024-03-24T10:35:30.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 -- start start 2024-03-24T10:35:30.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 0x7fd9e80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 0x7fd9e80d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e815e770 con 0x7fd9e80b3320 2024-03-24T10:35:30.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e80b3190 con 0x7fd9e815e390 2024-03-24T10:35:30.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.661+0000 7fd9ecec9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e80d4e40 con 0x7fd9e80d1000 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9e659c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 0x7fd9e80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 0x7fd9e80d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9e6d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9e80b3320 0x7fd9e80cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:52186/0 (socket says 172.21.15.1:52186) 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9e6d9d640 1 -- 172.21.15.1:0/4284857082 learned_addr learned my addr 172.21.15.1:0/4284857082 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9dffff640 1 -- 172.21.15.1:0/4284857082 >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 msgr2=0x7fd9e80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9dffff640 1 --2- 172.21.15.1:0/4284857082 >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 0x7fd9e80d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9dffff640 1 -- 172.21.15.1:0/4284857082 >> v1:172.21.15.3:6789/0 conn(0x7fd9e80b3320 legacy=0x7fd9e80cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9dffff640 1 -- 172.21.15.1:0/4284857082 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e80d1e40 con 0x7fd9e80d1000 2024-03-24T10:35:30.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.662+0000 7fd9e659c640 1 --2- 172.21.15.1:0/4284857082 >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 0x7fd9e80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.663+0000 7fd9dffff640 1 --2- 172.21.15.1:0/4284857082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 0x7fd9e80d29e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b78dd69fa3d6b3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.663+0000 7fd9e5d9b640 1 -- 172.21.15.1:0/4284857082 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d002f050 con 0x7fd9e80d1000 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.663+0000 7fd9e5d9b640 1 -- 172.21.15.1:0/4284857082 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0031700 con 0x7fd9e80d1000 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.663+0000 7fd9e5d9b640 1 -- 172.21.15.1:0/4284857082 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0031a90 con 0x7fd9e80d1000 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 -- 172.21.15.1:0/4284857082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 msgr2=0x7fd9e80d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/4284857082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 0x7fd9e80d29e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b7a0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 -- 172.21.15.1:0/4284857082 shutdown_connections 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/4284857082 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80d1000 0x7fd9e80d29e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/4284857082 >> v2:172.21.15.3:3300/0 conn(0x7fd9e815e390 0x7fd9e80d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 -- 172.21.15.1:0/4284857082 >> 172.21.15.1:0/4284857082 conn(0x7fd9e80d3ec0 msgr2=0x7fd9e80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 -- 172.21.15.1:0/4284857082 shutdown_connections 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.664+0000 7fd9ecec9640 1 -- 172.21.15.1:0/4284857082 wait complete. 2024-03-24T10:35:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.665+0000 7fd9ecec9640 1 Processor -- start 2024-03-24T10:35:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.665+0000 7fd9ecec9640 1 -- start start 2024-03-24T10:35:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80b3320 0x7fd9e815f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e815e390 0x7fd9e815a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e8161870 con 0x7fd9e80d1000 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e806bb50 con 0x7fd9e815e390 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9ecec9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e8160220 con 0x7fd9e80b3320 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50132/0 (socket says 172.21.15.1:50132) 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 -- 172.21.15.1:0/1583691762 learned_addr learned my addr 172.21.15.1:0/1583691762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9e6d9d640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e815e390 0x7fd9e815a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9e659c640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80b3320 0x7fd9e815f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 -- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80b3320 msgr2=0x7fd9e815f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80b3320 0x7fd9e815f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 -- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e815e390 msgr2=0x7fd9e815a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.666+0000 7fd9dffff640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e815e390 0x7fd9e815a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9dffff640 1 -- 172.21.15.1:0/1583691762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d8009400 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9dffff640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0036040 tx=0x7fd9d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0033930 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0033ab0 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d0009400 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0033e60 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.667+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e8160730 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.668+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e804f4e0 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.668+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd9d003a040 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.669+0000 7fd9deffd640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 0x7fd9c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.669+0000 7fd9e659c640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 0x7fd9c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.669+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fd9d00ab510 con 0x7fd9e80d1000 2024-03-24T10:35:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.669+0000 7fd9e659c640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 0x7fd9c806dcd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009c30 tx=0x7fd9d8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.673+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd9d0075f70 con 0x7fd9e80d1000 2024-03-24T10:35:30.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.858+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9e80b1a20 con 0x7fd9e80d1000 2024-03-24T10:35:30.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.859+0000 7fd9deffd640 1 -- 172.21.15.1:0/1583691762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7fd9d0079630 con 0x7fd9e80d1000 2024-03-24T10:35:30.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:30.865 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 msgr2=0x7fd9c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 0x7fd9c806dcd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009c30 tx=0x7fd9d8009190 comp rx=0 tx=0).stop 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 msgr2=0x7fd9e815fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0036040 tx=0x7fd9d00046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 shutdown_connections 2024-03-24T10:35:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd9c806b810 0x7fd9c806dcd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9e815e390 0x7fd9e815a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9e80d1000 0x7fd9e815fce0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 --2- 172.21.15.1:0/1583691762 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd9e80b3320 0x7fd9e815f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 >> 172.21.15.1:0/1583691762 conn(0x7fd9e80d3ec0 msgr2=0x7fd9e80cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 shutdown_connections 2024-03-24T10:35:30.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.861+0000 7fd9ecec9640 1 -- 172.21.15.1:0/1583691762 wait complete. 2024-03-24T10:35:30.874 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T10:35:30.874 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:30.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.974+0000 7f8337864640 1 Processor -- start 2024-03-24T10:35:30.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.974+0000 7f8337864640 1 -- start start 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8337864640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f833015e390 0x7f83300d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8337864640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83300b4ba0 0x7f83300d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8337864640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f833015e770 con 0x7f83300d1000 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8337864640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f83300b3190 con 0x7f83300b4ba0 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8337864640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f83300d4e40 con 0x7f833015e390 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83300b4ba0 0x7f83300d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f83300b4ba0 0x7f83300d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50146/0 (socket says 172.21.15.1:50146) 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 -- 172.21.15.1:0/1784879248 learned_addr learned my addr 172.21.15.1:0/1784879248 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335600640 1 --2- 172.21.15.1:0/1784879248 >> v2:172.21.15.3:3300/0 conn(0x7f833015e390 0x7f83300d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 -- 172.21.15.1:0/1784879248 >> v2:172.21.15.3:3300/0 conn(0x7f833015e390 msgr2=0x7f83300d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 --2- 172.21.15.1:0/1784879248 >> v2:172.21.15.3:3300/0 conn(0x7f833015e390 0x7f83300d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.975+0000 7f8335e01640 1 -- 172.21.15.1:0/1784879248 >> v1:172.21.15.3:6789/0 conn(0x7f83300d1000 legacy=0x7f83300cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.976+0000 7f8335e01640 1 -- 172.21.15.1:0/1784879248 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83300d1e40 con 0x7f83300b4ba0 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.976+0000 7f8335e01640 1 --2- 172.21.15.1:0/1784879248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300b4ba0 0x7f83300d29e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f83300d3040 tx=0x7f832802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7590da761bab48f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.976+0000 7f8327fff640 1 -- 172.21.15.1:0/1784879248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832802c9f0 con 0x7f83300b4ba0 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.976+0000 7f8327fff640 1 -- 172.21.15.1:0/1784879248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f832802cb50 con 0x7f83300b4ba0 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.976+0000 7f8327fff640 1 -- 172.21.15.1:0/1784879248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83280343f0 con 0x7f83300b4ba0 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 -- 172.21.15.1:0/1784879248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300b4ba0 msgr2=0x7f83300d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 --2- 172.21.15.1:0/1784879248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300b4ba0 0x7f83300d29e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f83300d3040 tx=0x7f832802be50 comp rx=0 tx=0).stop 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 -- 172.21.15.1:0/1784879248 shutdown_connections 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 --2- 172.21.15.1:0/1784879248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300b4ba0 0x7f83300d29e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 --2- 172.21.15.1:0/1784879248 >> v2:172.21.15.3:3300/0 conn(0x7f833015e390 0x7f83300d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 -- 172.21.15.1:0/1784879248 >> 172.21.15.1:0/1784879248 conn(0x7f83300d3ec0 msgr2=0x7f83300d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 -- 172.21.15.1:0/1784879248 shutdown_connections 2024-03-24T10:35:30.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.977+0000 7f8337864640 1 -- 172.21.15.1:0/1784879248 wait complete. 2024-03-24T10:35:30.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.978+0000 7f8337864640 1 Processor -- start 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.978+0000 7f8337864640 1 -- start start 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83300b4ba0 0x7f8330155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f833015e390 0x7f83300cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8330155d80 con 0x7f83300d1000 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f83300cc760 con 0x7f83300b4ba0 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8337864640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8330152b40 con 0x7f833015e390 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8334dff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8334dff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50150/0 (socket says 172.21.15.1:50150) 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8334dff640 1 -- 172.21.15.1:0/3620215287 learned_addr learned my addr 172.21.15.1:0/3620215287 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8335600640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83300b4ba0 0x7f8330155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.979+0000 7f8335e01640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f833015e390 0x7f83300cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 -- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f833015e390 msgr2=0x7f83300cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f833015e390 0x7f83300cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 -- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83300b4ba0 msgr2=0x7f8330155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83300b4ba0 0x7f8330155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 -- 172.21.15.1:0/3620215287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8328009b30 con 0x7f83300d1000 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8334dff640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f832000c440 tx=0x7f8320001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200052d0 con 0x7f83300d1000 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8320005430 con 0x7f83300d1000 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83300cbb50 con 0x7f83300d1000 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320007260 con 0x7f83300d1000 2024-03-24T10:35:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.980+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83300cc110 con 0x7f83300d1000 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.981+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83300cc8f0 con 0x7f83300d1000 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.982+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f83200073c0 con 0x7f83300d1000 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.982+0000 7f8325ffb640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 0x7f831406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.982+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f8320089530 con 0x7f83300d1000 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.982+0000 7f8335600640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 0x7f831406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.983+0000 7f8335600640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 0x7f831406dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f832c0079a0 tx=0x7f832c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:30.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:30.986+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8320053e70 con 0x7f83300d1000 2024-03-24T10:35:31.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.203+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f833015e770 con 0x7f83300d1000 2024-03-24T10:35:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.204+0000 7f8325ffb640 1 -- 172.21.15.1:0/3620215287 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5165 (secure 0 0 0) 0x7f833015e770 con 0x7f83300d1000 2024-03-24T10:35:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:31.205 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:35:02.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:35:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 msgr2=0x7f831406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 0x7f831406dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f832c0079a0 tx=0x7f832c008040 comp rx=0 tx=0).stop 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 msgr2=0x7f83300caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f832000c440 tx=0x7f8320001e70 comp rx=0 tx=0).stop 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 shutdown_connections 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f831406b810 0x7f831406dcd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f833015e390 0x7f83300cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f83300d1000 0x7f83300caf90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 --2- 172.21.15.1:0/3620215287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83300b4ba0 0x7f8330155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 >> 172.21.15.1:0/3620215287 conn(0x7f83300d3ec0 msgr2=0x7f83300b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 shutdown_connections 2024-03-24T10:35:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.206+0000 7f8337864640 1 -- 172.21.15.1:0/3620215287 wait complete. 2024-03-24T10:35:31.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:31.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.319+0000 7fbafc0c3640 1 Processor -- start 2024-03-24T10:35:31.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 -- start start 2024-03-24T10:35:31.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbaf415e390 0x7fbaf40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf40d1000 0x7fbaf40d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbaf40b3320 0x7fbaf40cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbaf40b3190 con 0x7fbaf40b3320 2024-03-24T10:35:31.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbaf415e770 con 0x7fbaf415e390 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.320+0000 7fbafc0c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbaf40d4e40 con 0x7fbaf40d1000 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbaf40b3320 0x7fbaf40cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbaf40b3320 0x7fbaf40cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50152/0 (socket says 172.21.15.1:50152) 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 -- 172.21.15.1:0/2026720856 learned_addr learned my addr 172.21.15.1:0/2026720856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbaf9e5f640 1 --2- 172.21.15.1:0/2026720856 >> v2:172.21.15.3:3300/0 conn(0x7fbaf415e390 0x7fbaf40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbaf965e640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf40d1000 0x7fbaf40d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 -- 172.21.15.1:0/2026720856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf40d1000 msgr2=0x7fbaf40d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf40d1000 0x7fbaf40d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 -- 172.21.15.1:0/2026720856 >> v2:172.21.15.3:3300/0 conn(0x7fbaf415e390 msgr2=0x7fbaf40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 --2- 172.21.15.1:0/2026720856 >> v2:172.21.15.3:3300/0 conn(0x7fbaf415e390 0x7fbaf40d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.321+0000 7fbafa660640 1 -- 172.21.15.1:0/2026720856 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf40d1e40 con 0x7fbaf40b3320 2024-03-24T10:35:31.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafa660640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf40cea60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40d3040 tx=0x7fbaec02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639c9bb66ef4d307 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbaf8e5d640 1 -- 172.21.15.1:0/2026720856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec02c950 con 0x7fbaf40b3320 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbaf8e5d640 1 -- 172.21.15.1:0/2026720856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec02cab0 con 0x7fbaf40b3320 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbaf8e5d640 1 -- 172.21.15.1:0/2026720856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec02ce40 con 0x7fbaf40b3320 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 -- 172.21.15.1:0/2026720856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 msgr2=0x7fbaf40cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf40cea60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40d3040 tx=0x7fbaec02bdb0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 -- 172.21.15.1:0/2026720856 shutdown_connections 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf40cea60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/2026720856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf40d1000 0x7fbaf40d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/2026720856 >> v2:172.21.15.3:3300/0 conn(0x7fbaf415e390 0x7fbaf40d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 -- 172.21.15.1:0/2026720856 >> 172.21.15.1:0/2026720856 conn(0x7fbaf40d3ec0 msgr2=0x7fbaf40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 -- 172.21.15.1:0/2026720856 shutdown_connections 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.322+0000 7fbafc0c3640 1 -- 172.21.15.1:0/2026720856 wait complete. 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 Processor -- start 2024-03-24T10:35:31.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 -- start start 2024-03-24T10:35:31.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40d1000 0x7fbaf40ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 0x7fbaf4159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf40cd330 con 0x7fbaf40b3320 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf409e2a0 con 0x7fbaf40d1000 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbafc0c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbaf40cc030 con 0x7fbaf415e390 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbaf9e5f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbaf9e5f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50156/0 (socket says 172.21.15.1:50156) 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.324+0000 7fbaf9e5f640 1 -- 172.21.15.1:0/3953990542 learned_addr learned my addr 172.21.15.1:0/3953990542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf965e640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40d1000 0x7fbaf40ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbafa660640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 0x7fbaf4159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 -- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 msgr2=0x7fbaf4159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 0x7fbaf4159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 -- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40d1000 msgr2=0x7fbaf40ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40d1000 0x7fbaf40ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 -- 172.21.15.1:0/3953990542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec009b30 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbafa660640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 0x7fbaf4159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaf9e5f640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000d5f0 tx=0x7fbaf000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00125f0 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf0012750 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf40cc310 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0012b00 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.325+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf40cc8d0 con 0x7fbaf40b3320 2024-03-24T10:35:31.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.326+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf40cca90 con 0x7fbaf40b3320 2024-03-24T10:35:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.327+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbaf0012c60 con 0x7fbaf40b3320 2024-03-24T10:35:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.327+0000 7fbaea7fc640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 0x7fbad806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.328+0000 7fbaf965e640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 0x7fbad806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.328+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7fbaf008a760 con 0x7fbaf40b3320 2024-03-24T10:35:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.328+0000 7fbaf965e640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 0x7fbad806dd00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4068660 tx=0x7fbae4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.332+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbaf0054ff0 con 0x7fbaf40b3320 2024-03-24T10:35:31.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.550+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaf406c180 con 0x7fbaf40b3320 2024-03-24T10:35:31.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.551+0000 7fbaea7fc640 1 -- 172.21.15.1:0/3953990542 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5165 (secure 0 0 0) 0x7fbaf00586b0 con 0x7fbaf40b3320 2024-03-24T10:35:31.552 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:31.552 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:34:53.709456+0000","modified":"2024-03-24T10:35:02.918016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1351159218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1351159218},{"type":"v1","addr":"172.21.15.1:6837","nonce":1351159218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":2}]} 2024-03-24T10:35:31.553 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-24T10:35:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 msgr2=0x7fbad806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 0x7fbad806dd00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4068660 tx=0x7fbae4008040 comp rx=0 tx=0).stop 2024-03-24T10:35:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 msgr2=0x7fbaf409dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000d5f0 tx=0x7fbaf000dab0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 shutdown_connections 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbad806b840 0x7fbad806dd00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbaf415e390 0x7fbaf4159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40d1000 0x7fbaf40ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 --2- 172.21.15.1:0/3953990542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf40b3320 0x7fbaf409dc10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 >> 172.21.15.1:0/3953990542 conn(0x7fbaf40d3ec0 msgr2=0x7fbaf406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 shutdown_connections 2024-03-24T10:35:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.553+0000 7fbafc0c3640 1 -- 172.21.15.1:0/3953990542 wait complete. 2024-03-24T10:35:31.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.666+0000 7f4fbbc55640 1 Processor -- start 2024-03-24T10:35:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.666+0000 7f4fbbc55640 1 -- start start 2024-03-24T10:35:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fbbc55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fbbc55640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d1000 0x7f4fb40b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fbbc55640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb415e770 con 0x7f4fb415e390 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fbbc55640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb40b3190 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fbbc55640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb40d4e40 con 0x7f4fb40d1000 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50158/0 (socket says 172.21.15.1:50158) 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 -- 172.21.15.1:0/1492309505 learned_addr learned my addr 172.21.15.1:0/1492309505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fba1f2640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d1000 0x7f4fb40b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 -- 172.21.15.1:0/1492309505 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d1000 msgr2=0x7f4fb40b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d1000 0x7f4fb40b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 -- 172.21.15.1:0/1492309505 >> v1:172.21.15.3:6789/0 conn(0x7f4fb415e390 legacy=0x7f4fb40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.668+0000 7f4fb91f0640 1 -- 172.21.15.1:0/1492309505 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb40d1e40 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.669+0000 7f4fb91f0640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009870 tx=0x7f4fa402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15839cae91658c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.669+0000 7f4fb89ef640 1 -- 172.21.15.1:0/1492309505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40317c0 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.669+0000 7f4fb89ef640 1 -- 172.21.15.1:0/1492309505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4031920 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.669+0000 7f4fb89ef640 1 -- 172.21.15.1:0/1492309505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4031cb0 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 -- 172.21.15.1:0/1492309505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb40d2bb0 msgr2=0x7f4fb40b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009870 tx=0x7f4fa402b860 comp rx=0 tx=0).stop 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 -- 172.21.15.1:0/1492309505 shutdown_connections 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d1000 0x7f4fb40b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/1492309505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb40d2bb0 0x7f4fb40b3b90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 -- 172.21.15.1:0/1492309505 >> 172.21.15.1:0/1492309505 conn(0x7f4fb40d3ec0 msgr2=0x7f4fb40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 -- 172.21.15.1:0/1492309505 shutdown_connections 2024-03-24T10:35:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.670+0000 7f4fbbc55640 1 -- 172.21.15.1:0/1492309505 wait complete. 2024-03-24T10:35:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.671+0000 7f4fbbc55640 1 Processor -- start 2024-03-24T10:35:31.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.671+0000 7f4fbbc55640 1 -- start start 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.671+0000 7f4fbbc55640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fb40d1000 0x7f4fb4155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.671+0000 7f4fbbc55640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d2bb0 0x7f4fb41589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.671+0000 7f4fbbc55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fbbc55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb41f72a0 con 0x7f4fb415e390 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fbbc55640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb414f7c0 con 0x7f4fb40d1000 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fbbc55640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb41f74f0 con 0x7f4fb40d2bb0 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50170/0 (socket says 172.21.15.1:50170) 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 -- 172.21.15.1:0/2305151009 learned_addr learned my addr 172.21.15.1:0/2305151009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fb91f0640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d2bb0 0x7f4fb41589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fb99f1640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fb40d1000 0x7f4fb4155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 -- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d2bb0 msgr2=0x7f4fb41589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d2bb0 0x7f4fb41589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 -- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fb40d1000 msgr2=0x7f4fb4155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fb40d1000 0x7f4fb4155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 -- 172.21.15.1:0/2305151009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa40094d0 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fba1f2640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4fac007e60 tx=0x7f4fac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac014620 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fac014780 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb41f77d0 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac014b30 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.672+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb41f7d90 con 0x7f4fb415e390 2024-03-24T10:35:31.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.674+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb41f7f80 con 0x7f4fb415e390 2024-03-24T10:35:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.674+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4fac014c90 con 0x7f4fb415e390 2024-03-24T10:35:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.675+0000 7f4faa7fc640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 0x7f4f9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.675+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7f4fac08acb0 con 0x7f4fb415e390 2024-03-24T10:35:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.675+0000 7f4fb99f1640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 0x7f4f9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.675+0000 7f4fb99f1640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 0x7f4f9806dcd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000f60 tx=0x7f4fb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.679+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4fac0555d0 con 0x7f4fb415e390 2024-03-24T10:35:31.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.863+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fb404f4e0 con 0x7f4fb415e390 2024-03-24T10:35:31.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.864+0000 7f4faa7fc640 1 -- 172.21.15.1:0/2305151009 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+18349 (secure 0 0 0) 0x7f4fac058c90 con 0x7f4fb415e390 2024-03-24T10:35:31.866 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:31.866 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":37,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:04.384072+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:34:51.593102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T10:34:52.680355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-24T10:35:02.227765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 msgr2=0x7f4f9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 0x7f4f9806dcd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000f60 tx=0x7f4fb0009040 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 msgr2=0x7f4fb414f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4fac007e60 tx=0x7f4fac00e4b0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 shutdown_connections 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4f9806b810 0x7f4f9806dcd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fb415e390 0x7f4fb414f280 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fb40d2bb0 0x7f4fb41589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 --2- 172.21.15.1:0/2305151009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fb40d1000 0x7f4fb4155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.866+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 >> 172.21.15.1:0/2305151009 conn(0x7f4fb40d3ec0 msgr2=0x7f4fb40d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.867+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 shutdown_connections 2024-03-24T10:35:31.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.867+0000 7f4fbbc55640 1 -- 172.21.15.1:0/2305151009 wait complete. 2024-03-24T10:35:31.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T10:35:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.983+0000 7febe1ac9640 1 Processor -- start 2024-03-24T10:35:31.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.983+0000 7febe1ac9640 1 -- start start 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 0x7febdc0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febdc0b3320 0x7febdc0cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7febdc0b3190 con 0x7febdc0b3320 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7febdc15e770 con 0x7febdc15e390 2024-03-24T10:35:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febe1ac9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0d4e40 con 0x7febdc0d1000 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febdbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febdc0b3320 0x7febdc0cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febdbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7febdc0b3320 0x7febdc0cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50180/0 (socket says 172.21.15.1:50180) 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.984+0000 7febdbfff640 1 -- 172.21.15.1:0/1228724801 learned_addr learned my addr 172.21.15.1:0/1228724801 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdb7fe640 1 --2- 172.21.15.1:0/1228724801 >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 0x7febdc0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdaffd640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdbfff640 1 -- 172.21.15.1:0/1228724801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 msgr2=0x7febdc0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdbfff640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdbfff640 1 -- 172.21.15.1:0/1228724801 >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 msgr2=0x7febdc0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdbfff640 1 --2- 172.21.15.1:0/1228724801 >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 0x7febdc0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdbfff640 1 -- 172.21.15.1:0/1228724801 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc0d1e40 con 0x7febdc0b3320 2024-03-24T10:35:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.985+0000 7febdb7fe640 1 --2- 172.21.15.1:0/1228724801 >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 0x7febdc0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:31.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.986+0000 7febdbfff640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc0b3320 0x7febdc0cea60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7febdc0d3040 tx=0x7febd402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7987b345922e850 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.986+0000 7febda7fc640 1 -- 172.21.15.1:0/1228724801 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd402c950 con 0x7febdc0b3320 2024-03-24T10:35:31.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.986+0000 7febda7fc640 1 -- 172.21.15.1:0/1228724801 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd402cab0 con 0x7febdc0b3320 2024-03-24T10:35:31.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.986+0000 7febda7fc640 1 -- 172.21.15.1:0/1228724801 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd402ce40 con 0x7febdc0b3320 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 -- 172.21.15.1:0/1228724801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc0b3320 msgr2=0x7febdc0cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc0b3320 0x7febdc0cea60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7febdc0d3040 tx=0x7febd402bdb0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 -- 172.21.15.1:0/1228724801 shutdown_connections 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc0b3320 0x7febdc0cea60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1228724801 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1228724801 >> v2:172.21.15.3:3300/0 conn(0x7febdc15e390 0x7febdc0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 -- 172.21.15.1:0/1228724801 >> 172.21.15.1:0/1228724801 conn(0x7febdc0d3ec0 msgr2=0x7febdc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 -- 172.21.15.1:0/1228724801 shutdown_connections 2024-03-24T10:35:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.987+0000 7febe1ac9640 1 -- 172.21.15.1:0/1228724801 wait complete. 2024-03-24T10:35:31.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.988+0000 7febe1ac9640 1 Processor -- start 2024-03-24T10:35:31.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.988+0000 7febe1ac9640 1 -- start start 2024-03-24T10:35:31.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7febdc0b3320 0x7febdc1f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7febdc156240 con 0x7febdc15e390 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7febdc0cc550 con 0x7febdc0b3320 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febe1ac9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0cc830 con 0x7febdc0d1000 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50184/0 (socket says 172.21.15.1:50184) 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 -- 172.21.15.1:0/1845538494 learned_addr learned my addr 172.21.15.1:0/1845538494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdb7fe640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7febdc0b3320 0x7febdc1f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdaffd640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 -- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 msgr2=0x7febdc155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 -- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7febdc0b3320 msgr2=0x7febdc1f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7febdc0b3320 0x7febdc1f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.989+0000 7febdbfff640 1 -- 172.21.15.1:0/1845538494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd4009b30 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febdbfff640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7febd402bdf0 tx=0x7febd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd403e450 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd403e5b0 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc0ccb10 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd403e960 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.990+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc0a1b40 con 0x7febdc15e390 2024-03-24T10:35:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.991+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febdc0cd030 con 0x7febdc15e390 2024-03-24T10:35:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.992+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febd403eac0 con 0x7febdc15e390 2024-03-24T10:35:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.992+0000 7febd3fff640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 0x7febbc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.992+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6104+0+0 (secure 0 0 0) 0x7febd40ad0c0 con 0x7febdc15e390 2024-03-24T10:35:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.992+0000 7febdb7fe640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 0x7febbc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.993+0000 7febdb7fe640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 0x7febbc06dcd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7febc400aa40 tx=0x7febc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:31.997+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febdc0cd030 con 0x7febdc15e390 2024-03-24T10:35:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.316+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7febdc0cd030 con 0x7febdc15e390 2024-03-24T10:35:32.318 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-24T10:35:32.317+0000 7f388d532640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:35:32.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.318+0000 7febd3fff640 1 -- 172.21.15.1:0/1845538494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7febdc0cd030 con 0x7febdc15e390 2024-03-24T10:35:32.319 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 msgr2=0x7febbc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 0x7febbc06dcd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7febc400aa40 tx=0x7febc4008040 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 msgr2=0x7febdc0a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7febd402bdf0 tx=0x7febd4004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 shutdown_connections 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febbc06b810 0x7febbc06dcd0 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7febc400aa40 tx=0x7febc4008040 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febdc15e390 0x7febdc0a1240 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7febdc0d1000 0x7febdc155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 --2- 172.21.15.1:0/1845538494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7febdc0b3320 0x7febdc1f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.321+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 >> 172.21.15.1:0/1845538494 conn(0x7febdc0d3ec0 msgr2=0x7febdc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.322+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 shutdown_connections 2024-03-24T10:35:32.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.322+0000 7febe1ac9640 1 -- 172.21.15.1:0/1845538494 wait complete. 2024-03-24T10:35:32.325 INFO:tasks.ceph.mds.c.smithi001.stderr: -3547> 2024-03-24T10:34:58.662+0000 7fd6a8dda640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:35:32.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T10:35:32.358 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.441+0000 7f81c2cc9640 1 Processor -- start 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.441+0000 7f81c2cc9640 1 -- start start 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c2cc9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc15bed0 0x7f81bc0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c2cc9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81bc0b5880 0x7f81bc0b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c2cc9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81bc0ce250 con 0x7f81bc0afb00 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c2cc9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f81bc15c2b0 con 0x7f81bc0b5880 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c2cc9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f81bc0b6250 con 0x7f81bc15bed0 2024-03-24T10:35:32.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81bc0b5880 0x7f81bc0b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81bc0b5880 0x7f81bc0b5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50204/0 (socket says 172.21.15.1:50204) 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81bbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f81bc0afb00 0x7f81bc15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38484/0 (socket says 172.21.15.1:38484) 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 -- 172.21.15.1:0/857716005 learned_addr learned my addr 172.21.15.1:0/857716005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c0a65640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc15bed0 0x7f81bc0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 -- 172.21.15.1:0/857716005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc15bed0 msgr2=0x7f81bc0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc15bed0 0x7f81bc0cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.442+0000 7f81c1266640 1 -- 172.21.15.1:0/857716005 >> v1:172.21.15.1:6789/0 conn(0x7f81bc0afb00 legacy=0x7f81bc15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.443+0000 7f81c1266640 1 -- 172.21.15.1:0/857716005 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81bc0ce530 con 0x7f81bc0b5880 2024-03-24T10:35:32.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.443+0000 7f81c1266640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc0b5880 0x7f81bc0b5d10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009870 tx=0x7f81a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809781cdb339fd59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.443+0000 7f81bb7fe640 1 -- 172.21.15.1:0/857716005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4030840 con 0x7f81bc0b5880 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.443+0000 7f81bb7fe640 1 -- 172.21.15.1:0/857716005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a40309a0 con 0x7f81bc0b5880 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.443+0000 7f81bb7fe640 1 -- 172.21.15.1:0/857716005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4030d30 con 0x7f81bc0b5880 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 -- 172.21.15.1:0/857716005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc0b5880 msgr2=0x7f81bc0b5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc0b5880 0x7f81bc0b5d10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009870 tx=0x7f81a402b980 comp rx=0 tx=0).stop 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 -- 172.21.15.1:0/857716005 shutdown_connections 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc0b5880 0x7f81bc0b5d10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/857716005 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc15bed0 0x7f81bc0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 -- 172.21.15.1:0/857716005 >> 172.21.15.1:0/857716005 conn(0x7f81bc072410 msgr2=0x7f81bc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 -- 172.21.15.1:0/857716005 shutdown_connections 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.444+0000 7f81c2cc9640 1 -- 172.21.15.1:0/857716005 wait complete. 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.445+0000 7f81c2cc9640 1 Processor -- start 2024-03-24T10:35:32.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.445+0000 7f81c2cc9640 1 -- start start 2024-03-24T10:35:32.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc0afb00 0x7f81bc1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 0x7f81bc158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc15ef60 con 0x7f81bc15bed0 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc1554d0 con 0x7f81bc0b5880 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c2cc9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f81bc155770 con 0x7f81bc0afb00 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50208/0 (socket says 172.21.15.1:50208) 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 -- 172.21.15.1:0/3814424447 learned_addr learned my addr 172.21.15.1:0/3814424447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:32.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81bbfff640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 0x7f81bc158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c0a65640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc0afb00 0x7f81bc1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 -- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc0afb00 msgr2=0x7f81bc1588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc0afb00 0x7f81bc1588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 -- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 msgr2=0x7f81bc158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 0x7f81bc158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:32.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81c1266640 1 -- 172.21.15.1:0/3814424447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40094d0 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.446+0000 7f81bbfff640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 0x7f81bc158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81c1266640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f81a402b9c0 tx=0x7f81a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4039e20 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a4043400 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81bc15e9a0 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a40437b0 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.447+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81bc155a10 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.448+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81bc04f470 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.448+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f81a4043910 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.449+0000 7f81b97fa640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 0x7f819c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.449+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6190+0+0 (secure 0 0 0) 0x7f81a403e070 con 0x7f81bc15bed0 2024-03-24T10:35:32.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.449+0000 7f81c0a65640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 0x7f819c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.449+0000 7f81c0a65640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 0x7f819c06dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f81b400a970 tx=0x7f81b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:32.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.453+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f81a4078040 con 0x7f81bc15bed0 2024-03-24T10:35:32.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:32.673+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f81bc155c00 con 0x7f81bc15bed0 2024-03-24T10:35:33.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.322+0000 7f81b97fa640 1 -- 172.21.15.1:0/3814424447 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f81a407b700 con 0x7f81bc15bed0 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 msgr2=0x7f819c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 0x7f819c06dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f81b400a970 tx=0x7f81b4008040 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 msgr2=0x7f81bc15dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f81a402b9c0 tx=0x7f81a4004730 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 shutdown_connections 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f819c06b810 0x7f819c06dcd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81bc15bed0 0x7f81bc15dce0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81bc0b5880 0x7f81bc158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 --2- 172.21.15.1:0/3814424447 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f81bc0afb00 0x7f81bc1588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 >> 172.21.15.1:0/3814424447 conn(0x7f81bc072410 msgr2=0x7f81bc0ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 shutdown_connections 2024-03-24T10:35:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.325+0000 7f81c2cc9640 1 -- 172.21.15.1:0/3814424447 wait complete. 2024-03-24T10:35:33.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T10:35:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.491+0000 7fe2dc8c3640 1 Processor -- start 2024-03-24T10:35:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.491+0000 7fe2dc8c3640 1 -- start start 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2dc8c3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2d815e2c0 0x7fe2d806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2dc8c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2d806fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2dc8c3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d8070120 con 0x7fe2d80ce640 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2dc8c3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d80cc2b0 con 0x7fe2d815e2c0 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2dc8c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d80cc870 con 0x7fe2d806f780 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe2d80ce640 0x7fe2d80cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38494/0 (socket says 172.21.15.1:38494) 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d6d9d640 1 -- 172.21.15.1:0/1016684553 learned_addr learned my addr 172.21.15.1:0/1016684553 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d659c640 1 --2- 172.21.15.1:0/1016684553 >> v2:172.21.15.3:3300/0 conn(0x7fe2d815e2c0 0x7fe2d806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323368379 0 0) 0x7fe2d8070120 con 0x7fe2d80ce640 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d5d9b640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2d806fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.492+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2c4003520 con 0x7fe2d80ce640 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1360889885 0 0) 0x7fe2c4003520 con 0x7fe2d80ce640 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 msgr2=0x7fe2d806fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2d806fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 >> v2:172.21.15.3:3300/0 conn(0x7fe2d815e2c0 msgr2=0x7fe2d806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 --2- 172.21.15.1:0/1016684553 >> v2:172.21.15.3:3300/0 conn(0x7fe2d815e2c0 0x7fe2d806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d80d0d50 con 0x7fe2d80ce640 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fe2c80019f0 con 0x7fe2d80ce640 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 >> v1:172.21.15.1:6789/0 conn(0x7fe2d80ce640 legacy=0x7fe2d80cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c4003970 0x7fe2c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c40047c0 0x7fe2c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d80cc2b0 con 0x7fe2c4003970 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d80cc870 con 0x7fe2c40047c0 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c400b980 con 0x7fe2d806f780 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c4003970 0x7fe2c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d5d9b640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c40047c0 0x7fe2c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d6d9d640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 -- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 msgr2=0x7fe2c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2d806f780 0x7fe2c4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 -- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c40047c0 msgr2=0x7fe2c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c40047c0 0x7fe2c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.493+0000 7fe2d659c640 1 -- 172.21.15.1:0/1016684553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c4003520 con 0x7fe2c4003970 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.494+0000 7fe2d659c640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c4003970 0x7fe2c4003d90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009af0 tx=0x7fe2c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfeb534ae5e54f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.494+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d002c710 con 0x7fe2c4003970 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.494+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d002c870 con 0x7fe2c4003970 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.494+0000 7fe2d559a640 1 -- 172.21.15.1:0/1016684553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d002cc00 con 0x7fe2c4003970 2024-03-24T10:35:33.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/1016684553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c4003970 msgr2=0x7fe2c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/1016684553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c4003970 0x7fe2c4003d90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009af0 tx=0x7fe2c40057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2d6d9d640 1 -- 172.21.15.1:0/1016684553 reap_dead start 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/1016684553 shutdown_connections 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/1016684553 >> 172.21.15.1:0/1016684553 conn(0x7fe2d80d1c60 msgr2=0x7fe2d80d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/1016684553 shutdown_connections 2024-03-24T10:35:33.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.495+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/1016684553 wait complete. 2024-03-24T10:35:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.496+0000 7fe2dc8c3640 1 Processor -- start 2024-03-24T10:35:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.496+0000 7fe2dc8c3640 1 -- start start 2024-03-24T10:35:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 0x7fe2d806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d80caf00 0x7fe2d80cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d80cb8c0 con 0x7fe2d8153280 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d815f360 con 0x7fe2d80caf00 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2dc8c3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d8156c50 con 0x7fe2c4003970 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50224/0 (socket says 172.21.15.1:50224) 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 -- 172.21.15.1:0/349384472 learned_addr learned my addr 172.21.15.1:0/349384472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d6d9d640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d80caf00 0x7fe2d80cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d659c640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 0x7fe2d806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 -- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 msgr2=0x7fe2d806cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 0x7fe2d806cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 -- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d80caf00 msgr2=0x7fe2d80cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d80caf00 0x7fe2d80cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:33.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 -- 172.21.15.1:0/349384472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d81536e0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d659c640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 0x7fe2d806cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.497+0000 7fe2d5d9b640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0011940 tx=0x7fe2c0011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.498+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c000f9d0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.498+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c000fb30 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.498+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d8156ec0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.498+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c000fec0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.498+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d806d4f0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.499+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d804f4e0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.499+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe2c0014440 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.500+0000 7fe2ceffd640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 0x7fe2b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.500+0000 7fe2d659c640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 0x7fe2b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.500+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6190+0+0 (secure 0 0 0) 0x7fe2c008aad0 con 0x7fe2d8153280 2024-03-24T10:35:33.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.500+0000 7fe2d659c640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 0x7fe2b806dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0036000 tx=0x7fe2d0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:33.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.504+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe2c0055310 con 0x7fe2d8153280 2024-03-24T10:35:33.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:33.686+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe2d815f4a0 con 0x7fe2d8153280 2024-03-24T10:35:34.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.324+0000 7fe2ceffd640 1 -- 172.21.15.1:0/349384472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v39) ==== 181+0+0 (secure 0 0 0) 0x7fe2c00589d0 con 0x7fe2d8153280 2024-03-24T10:35:34.325 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-24T10:35:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.328+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 msgr2=0x7fe2b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.328+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 0x7fe2b806dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0036000 tx=0x7fe2d0030000 comp rx=0 tx=0).stop 2024-03-24T10:35:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.328+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 msgr2=0x7fe2d8153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.328+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0011940 tx=0x7fe2c0011d40 comp rx=0 tx=0).stop 2024-03-24T10:35:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 shutdown_connections 2024-03-24T10:35:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe2b806b810 0x7fe2b806dcd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d80caf00 0x7fe2d80cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2d8153280 0x7fe2d8153fa0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 --2- 172.21.15.1:0/349384472 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe2c4003970 0x7fe2d806cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.330+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 >> 172.21.15.1:0/349384472 conn(0x7fe2d80d1c60 msgr2=0x7fe2d80cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:34.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.331+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 shutdown_connections 2024-03-24T10:35:34.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.333+0000 7fe2dc8c3640 1 -- 172.21.15.1:0/349384472 wait complete. 2024-03-24T10:35:34.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T10:35:34.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.460+0000 7fb930fc2640 1 Processor -- start 2024-03-24T10:35:34.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.460+0000 7fb930fc2640 1 -- start start 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb930fc2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 0x7fb92c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb930fc2640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 0x7fb92c0d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb930fc2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c15f580 con 0x7fb92c0d1700 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb930fc2640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c15efc0 con 0x7fb92c15ead0 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb930fc2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c15f2a0 con 0x7fb92c0cffc0 2024-03-24T10:35:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb92ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb92c0d1700 0x7fb92c06ddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38496/0 (socket says 172.21.15.1:38496) 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.461+0000 7fb92ad9d640 1 -- 172.21.15.1:0/230392717 learned_addr learned my addr 172.21.15.1:0/230392717 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb92a59c640 1 --2- 172.21.15.1:0/230392717 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 0x7fb92c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb923fff640 1 --2- 172.21.15.1:0/230392717 >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 0x7fb92c0d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964060193 0 0) 0x7fb92c15f580 con 0x7fb92c0d1700 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb918003520 con 0x7fb92c0d1700 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb92a59c640 1 -- 172.21.15.1:0/230392717 >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 msgr2=0x7fb92c0d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb92a59c640 1 --2- 172.21.15.1:0/230392717 >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 0x7fb92c0d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb92a59c640 1 -- 172.21.15.1:0/230392717 >> v1:172.21.15.1:6789/0 conn(0x7fb92c0d1700 legacy=0x7fb92c06ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2515902570 0 0) 0x7fb918003520 con 0x7fb92c0d1700 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb92a59c640 1 -- 172.21.15.1:0/230392717 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c0d09c0 con 0x7fb92c0cffc0 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.462+0000 7fb923fff640 1 --2- 172.21.15.1:0/230392717 >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 0x7fb92c0d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb92a59c640 1 --2- 172.21.15.1:0/230392717 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 0x7fb92c15e590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0097a0 tx=0x7fb91c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9104da6169f5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240019f0 con 0x7fb92c0cffc0 2024-03-24T10:35:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb91c02e050 con 0x7fb92c0cffc0 2024-03-24T10:35:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb929d9b640 1 -- 172.21.15.1:0/230392717 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c030710 con 0x7fb92c0cffc0 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb930fc2640 1 -- 172.21.15.1:0/230392717 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 msgr2=0x7fb92c15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.463+0000 7fb930fc2640 1 --2- 172.21.15.1:0/230392717 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 0x7fb92c15e590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0097a0 tx=0x7fb91c02b710 comp rx=0 tx=0).stop 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 -- 172.21.15.1:0/230392717 shutdown_connections 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 --2- 172.21.15.1:0/230392717 >> v2:172.21.15.3:3300/0 conn(0x7fb92c15ead0 0x7fb92c0d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 --2- 172.21.15.1:0/230392717 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0cffc0 0x7fb92c15e590 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 -- 172.21.15.1:0/230392717 >> 172.21.15.1:0/230392717 conn(0x7fb92c06c1b0 msgr2=0x7fb92c06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 -- 172.21.15.1:0/230392717 shutdown_connections 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.464+0000 7fb930fc2640 1 -- 172.21.15.1:0/230392717 wait complete. 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.465+0000 7fb930fc2640 1 Processor -- start 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.465+0000 7fb930fc2640 1 -- start start 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0d1700 0x7fb92c161810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c15ead0 0x7fb92c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c142ee0 con 0x7fb92c0cffc0 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c158e00 con 0x7fb92c15ead0 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c1590a0 con 0x7fb92c0d1700 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50234/0 (socket says 172.21.15.1:50234) 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 -- 172.21.15.1:0/868804863 learned_addr learned my addr 172.21.15.1:0/868804863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:34.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92ad9d640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c15ead0 0x7fb92c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb923fff640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0d1700 0x7fb92c161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 -- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0d1700 msgr2=0x7fb92c161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0d1700 0x7fb92c161810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 -- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c15ead0 msgr2=0x7fb92c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c15ead0 0x7fb92c15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 -- 172.21.15.1:0/868804863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb914009400 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb92a59c640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb91c02b750 tx=0x7fb91c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c03b810 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb91c03b970 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91c009400 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.466+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c03bd20 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.467+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c159660 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.467+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92c06fc80 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.468+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb91c039040 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.468+0000 7fb922ffd640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 0x7fb90c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.468+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5752+0+0 (secure 0 0 0) 0x7fb91c0ab9e0 con 0x7fb92c0cffc0 2024-03-24T10:35:34.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.468+0000 7fb923fff640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 0x7fb90c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.469+0000 7fb923fff640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 0x7fb90c06dd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb92c160280 tx=0x7fb914008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.473+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb91c076330 con 0x7fb92c0cffc0 2024-03-24T10:35:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:34.656+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb92c04f470 con 0x7fb92c0cffc0 2024-03-24T10:35:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.331+0000 7fb922ffd640 1 -- 172.21.15.1:0/868804863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v40) ==== 169+0+0 (secure 0 0 0) 0x7fb91c04b0d0 con 0x7fb92c0cffc0 2024-03-24T10:35:35.332 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-24T10:35:35.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.334+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 msgr2=0x7fb90c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.334+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 0x7fb90c06dd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb92c160280 tx=0x7fb914008040 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.334+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 msgr2=0x7fb92c1429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.334+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb91c02b750 tx=0x7fb91c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 shutdown_connections 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb90c06b840 0x7fb90c06dd00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb92c15ead0 0x7fb92c15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb92c0d1700 0x7fb92c161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 --2- 172.21.15.1:0/868804863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb92c0cffc0 0x7fb92c1429a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.335+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 >> 172.21.15.1:0/868804863 conn(0x7fb92c06c1b0 msgr2=0x7fb92c06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:35.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.339+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 shutdown_connections 2024-03-24T10:35:35.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.341+0000 7fb930fc2640 1 -- 172.21.15.1:0/868804863 wait complete. 2024-03-24T10:35:35.358 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-24T10:35:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.466+0000 7f6a2ccc9640 1 Processor -- start 2024-03-24T10:35:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.466+0000 7f6a2ccc9640 1 -- start start 2024-03-24T10:35:35.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a2ccc9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a280b4570 0x7f6a280b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a2ccc9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b2f50 0x7f6a2815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a2ccc9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a280b34d0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a2ccc9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6a280b37b0 con 0x7f6a280b4570 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a2ccc9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6a280b3a90 con 0x7f6a280b2f50 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a280b4570 0x7f6a280b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a280b4570 0x7f6a280b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50250/0 (socket says 172.21.15.1:50250) 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 -- 172.21.15.1:0/80760819 learned_addr learned my addr 172.21.15.1:0/80760819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a26d9d640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b2f50 0x7f6a2815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 -- 172.21.15.1:0/80760819 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b2f50 msgr2=0x7f6a2815c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b2f50 0x7f6a2815c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 -- 172.21.15.1:0/80760819 >> v1:172.21.15.3:6789/0 conn(0x7f6a2806e5b0 legacy=0x7f6a2806e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.467+0000 7f6a25d9b640 1 -- 172.21.15.1:0/80760819 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a280b3d70 con 0x7f6a280b4570 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.468+0000 7f6a25d9b640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a280b4570 0x7f6a280b2a10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6a100098a0 tx=0x7f6a1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c1f48620349b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.468+0000 7f6a2559a640 1 -- 172.21.15.1:0/80760819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a10030840 con 0x7f6a280b4570 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.468+0000 7f6a2559a640 1 -- 172.21.15.1:0/80760819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a100309a0 con 0x7f6a280b4570 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.468+0000 7f6a2559a640 1 -- 172.21.15.1:0/80760819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a10030d30 con 0x7f6a280b4570 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/80760819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a280b4570 msgr2=0x7f6a280b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a280b4570 0x7f6a280b2a10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6a100098a0 tx=0x7f6a1002b980 comp rx=0 tx=0).stop 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/80760819 shutdown_connections 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b2f50 0x7f6a2815c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/80760819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a280b4570 0x7f6a280b2a10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/80760819 >> 172.21.15.1:0/80760819 conn(0x7f6a280b1710 msgr2=0x7f6a280b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/80760819 shutdown_connections 2024-03-24T10:35:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.469+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/80760819 wait complete. 2024-03-24T10:35:35.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.470+0000 7f6a2ccc9640 1 Processor -- start 2024-03-24T10:35:35.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.470+0000 7f6a2ccc9640 1 -- start start 2024-03-24T10:35:35.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a280b2f50 0x7f6a280cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b4570 0x7f6a280cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2813be20 con 0x7f6a2806e5b0 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281f18b0 con 0x7f6a280b2f50 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2ccc9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6a280d0370 con 0x7f6a280b4570 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50260/0 (socket says 172.21.15.1:50260) 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 -- 172.21.15.1:0/274014259 learned_addr learned my addr 172.21.15.1:0/274014259 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 -- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b4570 msgr2=0x7f6a280cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a26d9d640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b4570 0x7f6a280cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a25d9b640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a280b2f50 0x7f6a280cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b4570 0x7f6a280cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 -- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a280b2f50 msgr2=0x7f6a280cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a280b2f50 0x7f6a280cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 -- 172.21.15.1:0/274014259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a100094d0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.471+0000 7f6a2659c640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c430 tx=0x7f6a2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.472+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a200052d0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.472+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a20005450 con 0x7f6a2806e5b0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.472+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20007260 con 0x7f6a2806e5b0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.472+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a28155cc0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.472+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2813bb40 con 0x7f6a2806e5b0 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.473+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2804f4e0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.473+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a200073c0 con 0x7f6a2806e5b0 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.474+0000 7f6a1effd640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 0x7f6a0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.474+0000 7f6a25d9b640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 0x7f6a0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.474+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5338+0+0 (secure 0 0 0) 0x7f6a20089410 con 0x7f6a2806e5b0 2024-03-24T10:35:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.474+0000 7f6a25d9b640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 0x7f6a0806dcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6a10036000 tx=0x7f6a1003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:35.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.478+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a20054030 con 0x7f6a2806e5b0 2024-03-24T10:35:35.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:35.661+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6a28143c60 con 0x7f6a2806e5b0 2024-03-24T10:35:36.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.332+0000 7f6a1effd640 1 -- 172.21.15.1:0/274014259 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v41) ==== 160+0+0 (secure 0 0 0) 0x7f6a200576f0 con 0x7f6a2806e5b0 2024-03-24T10:35:36.334 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'data_new' removed 2024-03-24T10:35:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.336+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 msgr2=0x7f6a0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.336+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 0x7f6a0806dcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6a10036000 tx=0x7f6a1003b040 comp rx=0 tx=0).stop 2024-03-24T10:35:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.336+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 msgr2=0x7f6a28155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.336+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c430 tx=0x7f6a2000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.337+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 shutdown_connections 2024-03-24T10:35:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6a0806b810 0x7f6a0806dcd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6a280b4570 0x7f6a280cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a280b2f50 0x7f6a280cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 --2- 172.21.15.1:0/274014259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2806e5b0 0x7f6a28155780 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 >> 172.21.15.1:0/274014259 conn(0x7f6a280b1710 msgr2=0x7f6a280af640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 shutdown_connections 2024-03-24T10:35:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.338+0000 7f6a2ccc9640 1 -- 172.21.15.1:0/274014259 wait complete. 2024-03-24T10:35:36.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T10:35:36.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.461+0000 7eff1dd43640 1 Processor -- start 2024-03-24T10:35:36.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.461+0000 7eff1dd43640 1 -- start start 2024-03-24T10:35:36.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff1dd43640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff18144b10 0x7eff18072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff1dd43640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff180cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff1dd43640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff18073050 con 0x7eff1815fa30 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff1dd43640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7eff180cf250 con 0x7eff18144b10 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff1dd43640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7eff18145a10 con 0x7eff1806b7b0 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff18144b10 0x7eff18072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff18144b10 0x7eff18072b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50268/0 (socket says 172.21.15.1:50268) 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff16ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7eff1815fa30 0x7eff1806b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38504/0 (socket says 172.21.15.1:38504) 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 -- 172.21.15.1:0/70438603 learned_addr learned my addr 172.21.15.1:0/70438603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff17fff640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff180cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 -- 172.21.15.1:0/70438603 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 msgr2=0x7eff180cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff180cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 -- 172.21.15.1:0/70438603 >> v1:172.21.15.1:6789/0 conn(0x7eff1815fa30 legacy=0x7eff1806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff167fc640 1 -- 172.21.15.1:0/70438603 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674834615 0 0) 0x7eff18073050 con 0x7eff1815fa30 2024-03-24T10:35:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.462+0000 7eff177fe640 1 -- 172.21.15.1:0/70438603 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1815fec0 con 0x7eff18144b10 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.463+0000 7eff177fe640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff18144b10 0x7eff18072b10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e28360d8b4a93b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.463+0000 7eff167fc640 1 -- 172.21.15.1:0/70438603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08030840 con 0x7eff18144b10 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.463+0000 7eff167fc640 1 -- 172.21.15.1:0/70438603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff080309a0 con 0x7eff18144b10 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.463+0000 7eff167fc640 1 -- 172.21.15.1:0/70438603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08030d30 con 0x7eff18144b10 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 -- 172.21.15.1:0/70438603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff18144b10 msgr2=0x7eff18072b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff18144b10 0x7eff18072b10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).stop 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 -- 172.21.15.1:0/70438603 shutdown_connections 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff180cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 --2- 172.21.15.1:0/70438603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff18144b10 0x7eff18072b10 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 -- 172.21.15.1:0/70438603 >> 172.21.15.1:0/70438603 conn(0x7eff1806c1b0 msgr2=0x7eff1806c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:36.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 -- 172.21.15.1:0/70438603 shutdown_connections 2024-03-24T10:35:36.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.464+0000 7eff1dd43640 1 -- 172.21.15.1:0/70438603 wait complete. 2024-03-24T10:35:36.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.465+0000 7eff1dd43640 1 Processor -- start 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.465+0000 7eff1dd43640 1 -- start start 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff1815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 0x7eff180c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff1815bde0 con 0x7eff181f1b10 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7eff180c7fc0 con 0x7eff18144b10 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff1dd43640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7eff180c73b0 con 0x7eff1806b7b0 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50278/0 (socket says 172.21.15.1:50278) 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 -- 172.21.15.1:0/4253527742 learned_addr learned my addr 172.21.15.1:0/4253527742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff177fe640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff1815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff16ffd640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 0x7eff180c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 -- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 msgr2=0x7eff1815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff1815b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 -- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 msgr2=0x7eff180c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 0x7eff180c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 -- 172.21.15.1:0/4253527742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff080094d0 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff16ffd640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 0x7eff180c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff177fe640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff1815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.466+0000 7eff17fff640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7eff1000efc0 tx=0x7eff1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.467+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10014480 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.467+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff1000f040 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.467+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff180c7690 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.467+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10014820 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.467+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff180c7c50 con 0x7eff181f1b10 2024-03-24T10:35:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.468+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff180c8170 con 0x7eff181f1b10 2024-03-24T10:35:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.468+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff10014980 con 0x7eff181f1b10 2024-03-24T10:35:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.469+0000 7efefffff640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 0x7eff0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.469+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7eff1008add0 con 0x7eff181f1b10 2024-03-24T10:35:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.469+0000 7eff177fe640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 0x7eff0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.469+0000 7eff177fe640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 0x7eff0406dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.474+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff180c8170 con 0x7eff181f1b10 2024-03-24T10:35:36.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.655+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7eff180c8130 con 0x7eff181f1b10 2024-03-24T10:35:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.658+0000 7efefffff640 1 -- 172.21.15.1:0/4253527742 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7eff10055af0 con 0x7eff181f1b10 2024-03-24T10:35:36.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 msgr2=0x7eff0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 0x7eff0406dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 msgr2=0x7eff180cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7eff1000efc0 tx=0x7eff1000c490 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 shutdown_connections 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7eff0406b810 0x7eff0406dcd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff181f1b10 0x7eff180cb8b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff18144b10 0x7eff180c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 --2- 172.21.15.1:0/4253527742 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7eff1806b7b0 0x7eff1815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 >> 172.21.15.1:0/4253527742 conn(0x7eff1806c1b0 msgr2=0x7eff1815c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 shutdown_connections 2024-03-24T10:35:36.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.660+0000 7eff1dd43640 1 -- 172.21.15.1:0/4253527742 wait complete. 2024-03-24T10:35:36.672 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.772+0000 7f7e0890f640 1 Processor -- start 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.772+0000 7f7e0890f640 1 -- start start 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e0890f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e04072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e0890f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e0406c910 0x7f7e0406cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e0890f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e0415b9c0 con 0x7f7e041549e0 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e0890f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7e040d0690 con 0x7f7e0406c910 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e0890f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0406d2a0 con 0x7f7e0415b470 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e02d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e0406c910 0x7f7e0406cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e02d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e0406c910 0x7f7e0406cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50280/0 (socket says 172.21.15.1:50280) 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.773+0000 7f7e02d9d640 1 -- 172.21.15.1:0/2584266974 learned_addr learned my addr 172.21.15.1:0/2584266974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e02d9d640 1 -- 172.21.15.1:0/2584266974 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 msgr2=0x7f7e04072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e01d9b640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e04072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e02d9d640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e04072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e02d9d640 1 -- 172.21.15.1:0/2584266974 >> v1:172.21.15.3:6789/0 conn(0x7f7e041549e0 legacy=0x7f7e04071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e02d9d640 1 -- 172.21.15.1:0/2584266974 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e04070ca0 con 0x7f7e0406c910 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e02d9d640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e0406c910 0x7f7e0406cd60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009e50 tx=0x7f7df402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d9a1b6eeaa1e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.774+0000 7f7e0159a640 1 -- 172.21.15.1:0/2584266974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df402ca70 con 0x7f7e0406c910 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0159a640 1 -- 172.21.15.1:0/2584266974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df402cbd0 con 0x7f7e0406c910 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0159a640 1 -- 172.21.15.1:0/2584266974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df40343f0 con 0x7f7e0406c910 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 -- 172.21.15.1:0/2584266974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e0406c910 msgr2=0x7f7e0406cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e0406c910 0x7f7e0406cd60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009e50 tx=0x7f7df402bed0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 -- 172.21.15.1:0/2584266974 shutdown_connections 2024-03-24T10:35:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e0406c910 0x7f7e0406cd60 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2584266974 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e04072990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 -- 172.21.15.1:0/2584266974 >> 172.21.15.1:0/2584266974 conn(0x7f7e040b17a0 msgr2=0x7f7e040b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 -- 172.21.15.1:0/2584266974 shutdown_connections 2024-03-24T10:35:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.775+0000 7f7e0890f640 1 -- 172.21.15.1:0/2584266974 wait complete. 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.776+0000 7f7e0890f640 1 Processor -- start 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 -- start start 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0406c910 0x7f7e040b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e041f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041f42e0 con 0x7f7e041549e0 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0413fa40 con 0x7f7e0406c910 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e0890f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7e041f7e10 con 0x7f7e0415b470 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e01d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e01d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50282/0 (socket says 172.21.15.1:50282) 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.777+0000 7f7e01d9b640 1 -- 172.21.15.1:0/2610470412 learned_addr learned my addr 172.21.15.1:0/2610470412 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e02d9d640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e041f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e0259c640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0406c910 0x7f7e040b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 -- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 msgr2=0x7f7e041f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e041f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 -- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0406c910 msgr2=0x7f7e040b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0406c910 0x7f7e040b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 -- 172.21.15.1:0/2610470412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df4009b30 con 0x7f7e041549e0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e02d9d640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e041f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e01d9b640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c7f0 tx=0x7f7dec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec004220 con 0x7f7e041549e0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dec004380 con 0x7f7e041549e0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e041f8040 con 0x7f7e041549e0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec007840 con 0x7f7e041549e0 2024-03-24T10:35:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.778+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e041f8600 con 0x7f7e041549e0 2024-03-24T10:35:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.779+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e041f8820 con 0x7f7e041549e0 2024-03-24T10:35:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.779+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7dec007bb0 con 0x7f7e041549e0 2024-03-24T10:35:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.780+0000 7f7dfaffd640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 0x7f7de406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.780+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f7dec089dc0 con 0x7f7e041549e0 2024-03-24T10:35:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.780+0000 7f7e0259c640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 0x7f7de406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.780+0000 7f7e0259c640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 0x7f7de406dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7e0414eb60 tx=0x7f7dfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:36.785+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7dec054b70 con 0x7f7e041549e0 2024-03-24T10:35:37.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.040+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f7e0404f4e0 con 0x7f7e041549e0 2024-03-24T10:35:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.345+0000 7f7dfaffd640 1 -- 172.21.15.1:0/2610470412 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v121) ==== 147+0+0 (secure 0 0 0) 0x7f7dec01e020 con 0x7f7e041549e0 2024-03-24T10:35:37.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 msgr2=0x7f7de406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 0x7f7de406dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7e0414eb60 tx=0x7f7dfc009040 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 msgr2=0x7f7e0413f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c7f0 tx=0x7f7dec00ccb0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 shutdown_connections 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7de406b810 0x7f7de406dcd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7e0415b470 0x7f7e041f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e041549e0 0x7f7e0413f500 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 --2- 172.21.15.1:0/2610470412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e0406c910 0x7f7e040b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 >> 172.21.15.1:0/2610470412 conn(0x7f7e040b17a0 msgr2=0x7f7e040cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 shutdown_connections 2024-03-24T10:35:37.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.347+0000 7f7e0890f640 1 -- 172.21.15.1:0/2610470412 wait complete. 2024-03-24T10:35:37.360 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-24T10:35:37.360 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:35:37.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.465+0000 7f3f10481640 1 Processor -- start 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.465+0000 7f3f10481640 1 -- start start 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f10481640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f08144770 0x7f3f080d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f10481640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f080cdd60 con 0x7f3f080b2aa0 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f10481640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0806e200 con 0x7f3f08070b90 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f10481640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3f0806eaa0 con 0x7f3f08144770 2024-03-24T10:35:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0ea1e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3f080b2aa0 0x7f3f080ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38512/0 (socket says 172.21.15.1:38512) 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/792607460 learned_addr learned my addr 172.21.15.1:0/792607460 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0e21d640 1 --2- 172.21.15.1:0/792607460 >> v2:172.21.15.3:3300/0 conn(0x7f3f08144770 0x7f3f080d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242806440 0 0) 0x7f3f080cdd60 con 0x7f3f080b2aa0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ef4003520 con 0x7f3f080b2aa0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3440110227 0 0) 0x7f3ef4003520 con 0x7f3f080b2aa0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 >> v2:172.21.15.3:3300/0 conn(0x7f3f08144770 msgr2=0x7f3f080d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 --2- 172.21.15.1:0/792607460 >> v2:172.21.15.3:3300/0 conn(0x7f3f08144770 0x7f3f080d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 >> v1:172.21.15.3:6789/0 conn(0x7f3f08070b90 legacy=0x7f3f0806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0806e4e0 con 0x7f3f080b2aa0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f3f000019f0 con 0x7f3f080b2aa0 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 >> v1:172.21.15.1:6789/0 conn(0x7f3f080b2aa0 legacy=0x7f3f080ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ef4003970 0x7f3ef4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 0x7f3ef4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef4005180 0x7f3ef400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0806eaa0 con 0x7f3ef40047c0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef400c020 con 0x7f3ef4005180 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.466+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef400c1e0 con 0x7f3ef4003970 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 0x7f3ef4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0e21d640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ef4003970 0x7f3ef4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0ea1e640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef4005180 0x7f3ef400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 -- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ef4003970 msgr2=0x7f3ef4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3ef4003970 0x7f3ef4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 -- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef4005180 msgr2=0x7f3ef400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef4005180 0x7f3ef400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0da1c640 1 -- 172.21.15.1:0/792607460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef4003520 con 0x7f3ef40047c0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0ea1e640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef4005180 0x7f3ef400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.467+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/792607460 reap_dead start 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f0da1c640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 0x7f3ef4004c40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8001720 tx=0x7f3ef40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc48ea59bd029a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80052d0 con 0x7f3ef40047c0 2024-03-24T10:35:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef802e050 con 0x7f3ef40047c0 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f0d21b640 1 -- 172.21.15.1:0/792607460 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8003830 con 0x7f3ef40047c0 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 -- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 msgr2=0x7f3ef4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 0x7f3ef4004c40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8001720 tx=0x7f3ef40057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 -- 172.21.15.1:0/792607460 shutdown_connections 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 --2- 172.21.15.1:0/792607460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef40047c0 0x7f3ef4004c40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 -- 172.21.15.1:0/792607460 >> 172.21.15.1:0/792607460 conn(0x7f3f080b17a0 msgr2=0x7f3f080b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 -- 172.21.15.1:0/792607460 shutdown_connections 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.468+0000 7f3f10481640 1 -- 172.21.15.1:0/792607460 wait complete. 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.469+0000 7f3f10481640 1 Processor -- start 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 -- start start 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef40047c0 0x7f3f081369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3f0813c1e0 0x7f3f08142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08071040 con 0x7f3f08142cd0 2024-03-24T10:35:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08143160 con 0x7f3ef40047c0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f10481640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0815b960 con 0x7f3f0813c1e0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f0ea1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f0ea1e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50302/0 (socket says 172.21.15.1:50302) 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.470+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/3455327828 learned_addr learned my addr 172.21.15.1:0/3455327828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0e21d640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef40047c0 0x7f3f081369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0da1c640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3f0813c1e0 0x7f3f08142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3f0813c1e0 msgr2=0x7f3f08142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3f0813c1e0 0x7f3f08142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef40047c0 msgr2=0x7f3f081369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef40047c0 0x7f3f081369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 -- 172.21.15.1:0/3455327828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0813b140 con 0x7f3f08142cd0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f0ea1e640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3f00000f80 tx=0x7f3f0000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000eb30 con 0x7f3f08142cd0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f0000ecb0 con 0x7f3f08142cd0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0813b420 con 0x7f3f08142cd0 2024-03-24T10:35:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00014430 con 0x7f3f08142cd0 2024-03-24T10:35:37.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.471+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0813b9e0 con 0x7f3f08142cd0 2024-03-24T10:35:37.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.472+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0806edf0 con 0x7f3f08142cd0 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.472+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f00014590 con 0x7f3f08142cd0 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.473+0000 7f3efeffd640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 0x7f3eec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.473+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f3f00089bb0 con 0x7f3f08142cd0 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.473+0000 7f3f0e21d640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 0x7f3eec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.473+0000 7f3f0e21d640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 0x7f3eec06dd00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009b00 tx=0x7f3f04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.477+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f000584c0 con 0x7f3f08142cd0 2024-03-24T10:35:37.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.659+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f08071e70 con 0x7f3f08142cd0 2024-03-24T10:35:37.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.659+0000 7f3efeffd640 1 -- 172.21.15.1:0/3455327828 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+12886 (secure 0 0 0) 0x7f3f00058050 con 0x7f3f08142cd0 2024-03-24T10:35:37.660 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:37.660 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":41,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:36.332905+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6837/1351159218":"2024-03-25T10:35:32.318608+0000","172.21.15.1:6836/1351159218":"2024-03-25T10:35:32.318608+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 msgr2=0x7f3eec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 0x7f3eec06dd00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009b00 tx=0x7f3f04008040 comp rx=0 tx=0).stop 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 msgr2=0x7f3f0813a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3f00000f80 tx=0x7f3f0000e530 comp rx=0 tx=0).stop 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 shutdown_connections 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3eec06b840 0x7f3eec06dd00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f08142cd0 0x7f3f0813a560 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3f0813c1e0 0x7f3f08142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 --2- 172.21.15.1:0/3455327828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ef40047c0 0x7f3f081369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.661+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 >> 172.21.15.1:0/3455327828 conn(0x7f3f080b17a0 msgr2=0x7f3f08072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.662+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 shutdown_connections 2024-03-24T10:35:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.662+0000 7f3f10481640 1 -- 172.21.15.1:0/3455327828 wait complete. 2024-03-24T10:35:37.673 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:37.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.776+0000 7f4ae90f4640 1 Processor -- start 2024-03-24T10:35:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.776+0000 7f4ae90f4640 1 -- start start 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae90f4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ae415e390 0x7f4ae40d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae90f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae40b0b70 con 0x7f4ae40cfae0 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae90f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae40d1d60 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae90f4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4ae40b0de0 con 0x7f4ae415e390 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ae415e390 0x7f4ae40d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae2d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ae415e390 0x7f4ae40d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50318/0 (socket says 172.21.15.1:50318) 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae359e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4ae40cfae0 0x7f4ae40b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38526/0 (socket says 172.21.15.1:38526) 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.777+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/572383598 learned_addr learned my addr 172.21.15.1:0/572383598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.778+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/572383598 >> v1:172.21.15.3:6789/0 conn(0x7f4ae40b2ed0 legacy=0x7f4ae40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.778+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/572383598 >> v1:172.21.15.1:6789/0 conn(0x7f4ae40cfae0 legacy=0x7f4ae40b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.778+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/572383598 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450458418 0 0) 0x7f4ae40b0b70 con 0x7f4ae40cfae0 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.778+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/572383598 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae415e770 con 0x7f4ae415e390 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.778+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/572383598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae415e390 0x7f4ae40d2310 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009870 tx=0x7f4adc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fafe1434935fa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/572383598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc02f840 con 0x7f4ae415e390 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/572383598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc02f9a0 con 0x7f4ae415e390 2024-03-24T10:35:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae1d9b640 1 -- 172.21.15.1:0/572383598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc02fd30 con 0x7f4ae415e390 2024-03-24T10:35:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 -- 172.21.15.1:0/572383598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae415e390 msgr2=0x7f4ae40d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/572383598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae415e390 0x7f4ae40d2310 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009870 tx=0x7f4adc02ba20 comp rx=0 tx=0).stop 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 -- 172.21.15.1:0/572383598 shutdown_connections 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/572383598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae415e390 0x7f4ae40d2310 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 -- 172.21.15.1:0/572383598 >> 172.21.15.1:0/572383598 conn(0x7f4ae40d3ee0 msgr2=0x7f4ae40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 -- 172.21.15.1:0/572383598 shutdown_connections 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.779+0000 7f4ae90f4640 1 -- 172.21.15.1:0/572383598 wait complete. 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 Processor -- start 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 -- start start 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ae40cfae0 0x7f4ae40b9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4ae415e390 0x7f4ae40b9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae414e220 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae40ccbf0 con 0x7f4ae40cfae0 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.781+0000 7f4ae90f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae40cc910 con 0x7f4ae415e390 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50320/0 (socket says 172.21.15.1:50320) 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/3166962293 learned_addr learned my addr 172.21.15.1:0/3166962293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae259c640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ae40cfae0 0x7f4ae40b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae359e640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4ae415e390 0x7f4ae40b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4ae415e390 msgr2=0x7f4ae40b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4ae415e390 0x7f4ae40b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ae40cfae0 msgr2=0x7f4ae40b9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ae40cfae0 0x7f4ae40b9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 -- 172.21.15.1:0/3166962293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4adc0094d0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae2d9d640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0090b0 tx=0x7f4adc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc03c040 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc037d50 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae414dbd0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc004270 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.782+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae40a1a70 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.783+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae404f4e0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.784+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4adc0043d0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.784+0000 7f4acb7fe640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 0x7f4ad006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:37.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.784+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f4adc0ac2f0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.784+0000 7f4ae259c640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 0x7f4ad006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:37.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.785+0000 7f4ae259c640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 0x7f4ad006dcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40ce060 tx=0x7f4ad8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:37.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.788+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4adc0770c0 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.966+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ae40d3210 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.967+0000 7f4acb7fe640 1 -- 172.21.15.1:0/3166962293 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f4adc07a780 con 0x7f4ae40b2ed0 2024-03-24T10:35:37.967 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:37.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 msgr2=0x7f4ad006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 0x7f4ad006dcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40ce060 tx=0x7f4ad8009040 comp rx=0 tx=0).stop 2024-03-24T10:35:37.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 msgr2=0x7f4ae40a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0090b0 tx=0x7f4adc004730 comp rx=0 tx=0).stop 2024-03-24T10:35:37.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 shutdown_connections 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4ad006b810 0x7f4ad006dcd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4ae415e390 0x7f4ae40b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ae40cfae0 0x7f4ae40b9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 --2- 172.21.15.1:0/3166962293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ae40b2ed0 0x7f4ae40a1530 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 >> 172.21.15.1:0/3166962293 conn(0x7f4ae40d3ee0 msgr2=0x7f4ae40d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 shutdown_connections 2024-03-24T10:35:37.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:37.969+0000 7f4ae90f4640 1 -- 172.21.15.1:0/3166962293 wait complete. 2024-03-24T10:35:37.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-24T10:35:38.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.088+0000 7efe0f447640 1 Processor -- start 2024-03-24T10:35:38.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.088+0000 7efe0f447640 1 -- start start 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0f447640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe0815bb30 0x7efe080afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0f447640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efe0806b900 0x7efe0815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0f447640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0806bd30 con 0x7efe080affe0 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0f447640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efe0806dbf0 con 0x7efe0815bb30 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0f447640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7efe080b0470 con 0x7efe0806b900 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0d1e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe0815bb30 0x7efe080afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0c9e2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efe080affe0 0x7efe0806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38538/0 (socket says 172.21.15.1:38538) 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0c9e2640 1 -- 172.21.15.1:0/4260746929 learned_addr learned my addr 172.21.15.1:0/4260746929 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.089+0000 7efe0d9e4640 1 --2- 172.21.15.1:0/4260746929 >> v2:172.21.15.3:3300/0 conn(0x7efe0806b900 0x7efe0815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efe0d1e3640 1 -- 172.21.15.1:0/4260746929 >> v2:172.21.15.3:3300/0 conn(0x7efe0806b900 msgr2=0x7efe0815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efe0d1e3640 1 --2- 172.21.15.1:0/4260746929 >> v2:172.21.15.3:3300/0 conn(0x7efe0806b900 0x7efe0815e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efe0d1e3640 1 -- 172.21.15.1:0/4260746929 >> v1:172.21.15.1:6789/0 conn(0x7efe080affe0 legacy=0x7efe0806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:38.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efdfffff640 1 -- 172.21.15.1:0/4260746929 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575273736 0 0) 0x7efe0806bd30 con 0x7efe080affe0 2024-03-24T10:35:38.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efe0d1e3640 1 -- 172.21.15.1:0/4260746929 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0806bf60 con 0x7efe0815bb30 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efe0d1e3640 1 --2- 172.21.15.1:0/4260746929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0815bb30 0x7efe080afaa0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efe080b2740 tx=0x7efe0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e2d499285d6ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.090+0000 7efdfffff640 1 -- 172.21.15.1:0/4260746929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04031830 con 0x7efe0815bb30 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efdfffff640 1 -- 172.21.15.1:0/4260746929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe04031990 con 0x7efe0815bb30 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efdfffff640 1 -- 172.21.15.1:0/4260746929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04031d20 con 0x7efe0815bb30 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 -- 172.21.15.1:0/4260746929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0815bb30 msgr2=0x7efe080afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 --2- 172.21.15.1:0/4260746929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0815bb30 0x7efe080afaa0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efe080b2740 tx=0x7efe0402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 -- 172.21.15.1:0/4260746929 shutdown_connections 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 --2- 172.21.15.1:0/4260746929 >> v2:172.21.15.3:3300/0 conn(0x7efe0806b900 0x7efe0815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 --2- 172.21.15.1:0/4260746929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0815bb30 0x7efe080afaa0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 -- 172.21.15.1:0/4260746929 >> 172.21.15.1:0/4260746929 conn(0x7efe080b27b0 msgr2=0x7efe080b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 -- 172.21.15.1:0/4260746929 shutdown_connections 2024-03-24T10:35:38.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.091+0000 7efe0f447640 1 -- 172.21.15.1:0/4260746929 wait complete. 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.092+0000 7efe0f447640 1 Processor -- start 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.092+0000 7efe0f447640 1 -- start start 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efe080affe0 0x7efe081f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0815bb30 0x7efe08072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efe081f4c70 con 0x7efe0806b900 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efe0815f9e0 con 0x7efe0815bb30 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0f447640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efe080b1f70 con 0x7efe080affe0 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50328/0 (socket says 172.21.15.1:50328) 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 -- 172.21.15.1:0/72371885 learned_addr learned my addr 172.21.15.1:0/72371885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 -- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efe080affe0 msgr2=0x7efe081f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efe080affe0 0x7efe081f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 -- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0815bb30 msgr2=0x7efe08072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0815bb30 0x7efe08072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.093+0000 7efe0d1e3640 1 -- 172.21.15.1:0/72371885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe04009400 con 0x7efe0806b900 2024-03-24T10:35:38.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efe0d1e3640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efe04034040 tx=0x7efe040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04004230 con 0x7efe0806b900 2024-03-24T10:35:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe04004390 con 0x7efe0806b900 2024-03-24T10:35:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe080b2250 con 0x7efe0806b900 2024-03-24T10:35:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0403a870 con 0x7efe0806b900 2024-03-24T10:35:38.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.094+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0815f670 con 0x7efe0806b900 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.095+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe08075470 con 0x7efe0806b900 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.095+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe0403a9d0 con 0x7efe0806b900 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.096+0000 7efdfdffb640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 0x7efdec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.096+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7efe040abf10 con 0x7efe0806b900 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.096+0000 7efe0c9e2640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 0x7efdec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:38.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.096+0000 7efe0c9e2640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 0x7efdec06dd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efdf8007fd0 tx=0x7efdf8007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:38.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.100+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efe04076d50 con 0x7efe0806b900 2024-03-24T10:35:38.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:38.348+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7efe0815f4d0 con 0x7efe0806b900 2024-03-24T10:35:39.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.350+0000 7efdfdffb640 1 -- 172.21.15.1:0/72371885 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v123) ==== 137+0+0 (secure 0 0 0) 0x7efe0815f4d0 con 0x7efe0806b900 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.352+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 msgr2=0x7efdec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.352+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 0x7efdec06dd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efdf8007fd0 tx=0x7efdf8007920 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 msgr2=0x7efe0815bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efe04034040 tx=0x7efe040046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 shutdown_connections 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efdec06b840 0x7efdec06dd00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0815bb30 0x7efe08072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efe080affe0 0x7efe081f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 --2- 172.21.15.1:0/72371885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe0806b900 0x7efe0815bf10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 >> 172.21.15.1:0/72371885 conn(0x7efe080b27b0 msgr2=0x7efe08159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:39.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 shutdown_connections 2024-03-24T10:35:39.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.353+0000 7efe0f447640 1 -- 172.21.15.1:0/72371885 wait complete. 2024-03-24T10:35:39.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T10:35:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.476+0000 7f3459083640 1 Processor -- start 2024-03-24T10:35:39.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.476+0000 7f3459083640 1 -- start start 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.476+0000 7f3459083640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454144550 0x7f34540b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.476+0000 7f3459083640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345406c930 con 0x7f34540b3a60 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3459083640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345406cb80 con 0x7f3454071cc0 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3459083640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f34540b4310 con 0x7f3454144550 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3452d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454144550 0x7f34540b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f344bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f34540b3a60 0x7f34540ccfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38542/0 (socket says 172.21.15.1:38542) 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f344bfff640 1 -- 172.21.15.1:0/3071808628 learned_addr learned my addr 172.21.15.1:0/3071808628 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:39.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3452d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3454144550 0x7f34540b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50336/0 (socket says 172.21.15.1:50336) 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3452d9d640 1 -- 172.21.15.1:0/3071808628 >> v1:172.21.15.3:6789/0 conn(0x7f3454071cc0 legacy=0x7f345406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3452d9d640 1 -- 172.21.15.1:0/3071808628 >> v1:172.21.15.1:6789/0 conn(0x7f34540b3a60 legacy=0x7f34540ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f345259c640 1 -- 172.21.15.1:0/3071808628 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784656309 0 0) 0x7f345406c930 con 0x7f34540b3a60 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.477+0000 7f3452d9d640 1 -- 172.21.15.1:0/3071808628 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345406ce60 con 0x7f3454144550 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3452d9d640 1 --2- 172.21.15.1:0/3071808628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454144550 0x7f34540b3520 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f34440097a0 tx=0x7f344402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38f690a2a372fda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f345259c640 1 -- 172.21.15.1:0/3071808628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344402e070 con 0x7f3454144550 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f345259c640 1 -- 172.21.15.1:0/3071808628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34440315a0 con 0x7f3454144550 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f345259c640 1 -- 172.21.15.1:0/3071808628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444031930 con 0x7f3454144550 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 -- 172.21.15.1:0/3071808628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454144550 msgr2=0x7f34540b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 --2- 172.21.15.1:0/3071808628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454144550 0x7f34540b3520 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f34440097a0 tx=0x7f344402b840 comp rx=0 tx=0).stop 2024-03-24T10:35:39.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 -- 172.21.15.1:0/3071808628 shutdown_connections 2024-03-24T10:35:39.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 --2- 172.21.15.1:0/3071808628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3454144550 0x7f34540b3520 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 -- 172.21.15.1:0/3071808628 >> 172.21.15.1:0/3071808628 conn(0x7f345406b4f0 msgr2=0x7f345406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:39.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 -- 172.21.15.1:0/3071808628 shutdown_connections 2024-03-24T10:35:39.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.478+0000 7f3459083640 1 -- 172.21.15.1:0/3071808628 wait complete. 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 Processor -- start 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 -- start start 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 0x7f345415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3454144550 0x7f3454141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f345415c6a0 con 0x7f34540b3a60 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f345415ba20 con 0x7f3454071cc0 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.480+0000 7f3459083640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f345415bd00 con 0x7f3454144550 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50346/0 (socket says 172.21.15.1:50346) 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 -- 172.21.15.1:0/3848689517 learned_addr learned my addr 172.21.15.1:0/3848689517 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:39.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f3452d9d640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 0x7f345415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f345359e640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3454144550 0x7f3454141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 -- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3454144550 msgr2=0x7f3454141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3454144550 0x7f3454141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 -- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 msgr2=0x7f345415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 0x7f345415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 -- 172.21.15.1:0/3848689517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3444009400 con 0x7f34540b3a60 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f3452d9d640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 0x7f345415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344bfff640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c4d0 tx=0x7f343c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c0074e0 con 0x7f34540b3a60 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f343c007660 con 0x7f34540b3a60 2024-03-24T10:35:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345415bfe0 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c004a20 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.481+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3454141ca0 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.482+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f345415c7e0 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.483+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f343c004b80 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.483+0000 7f344b7fe640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 0x7f343406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.483+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f343c089450 con 0x7f34540b3a60 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.483+0000 7f3452d9d640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 0x7f343406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.484+0000 7f3452d9d640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 0x7f343406dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3444036000 tx=0x7f344403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.487+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f345415c7e0 con 0x7f34540b3a60 2024-03-24T10:35:39.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.668+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f345415c7e0 con 0x7f34540b3a60 2024-03-24T10:35:39.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.671+0000 7f344b7fe640 1 -- 172.21.15.1:0/3848689517 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f345415c7e0 con 0x7f34540b3a60 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 msgr2=0x7f343406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 0x7f343406dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3444036000 tx=0x7f344403d040 comp rx=0 tx=0).stop 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 msgr2=0x7f345415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c4d0 tx=0x7f343c00c990 comp rx=0 tx=0).stop 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 shutdown_connections 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f343406b810 0x7f343406dcd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3454144550 0x7f3454141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34540b3a60 0x7f345415b340 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 --2- 172.21.15.1:0/3848689517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3454071cc0 0x7f345415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 >> 172.21.15.1:0/3848689517 conn(0x7f345406b4f0 msgr2=0x7f34540cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:39.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 shutdown_connections 2024-03-24T10:35:39.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.673+0000 7f3459083640 1 -- 172.21.15.1:0/3848689517 wait complete. 2024-03-24T10:35:39.684 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T10:35:39.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:35:39.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:35:39.778 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T10:35:39.778 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:39.778 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:39.793 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:35:39.793 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:35:39.794 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:39.810 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:35:39.811 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:35:39.811 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:39.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:35:39.850 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:35:39.850 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:35:39.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:35:39.867 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T10:35:39.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.990+0000 7f0cf5e0b640 1 Processor -- start 2024-03-24T10:35:39.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 -- start start 2024-03-24T10:35:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 0x7f0cf00d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 0x7f0cf00d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cf015e770 con 0x7f0cf00d1000 2024-03-24T10:35:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f0cf00b3190 con 0x7f0cf015e390 2024-03-24T10:35:39.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0cf5e0b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf00d4e40 con 0x7f0cf00b4ba0 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0ceeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0cf00d1000 0x7f0cf00cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38548/0 (socket says 172.21.15.1:38548) 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.991+0000 7f0ceeffd640 1 -- 172.21.15.1:0/1761894225 learned_addr learned my addr 172.21.15.1:0/1761894225 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/1761894225 >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 0x7f0cf00d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0ceffff640 1 --2- 172.21.15.1:0/1761894225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 0x7f0cf00d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0cee7fc640 1 -- 172.21.15.1:0/1761894225 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948468762 0 0) 0x7f0ce4001110 con 0x7f0cf00d1000 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0cee7fc640 1 -- 172.21.15.1:0/1761894225 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cd8003520 con 0x7f0cf00d1000 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0ceffff640 1 -- 172.21.15.1:0/1761894225 >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 msgr2=0x7f0cf00d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0ceffff640 1 --2- 172.21.15.1:0/1761894225 >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 0x7f0cf00d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0ceffff640 1 -- 172.21.15.1:0/1761894225 >> v1:172.21.15.1:6789/0 conn(0x7f0cf00d1000 legacy=0x7f0cf00cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0ceffff640 1 -- 172.21.15.1:0/1761894225 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf00d1e40 con 0x7f0cf00b4ba0 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.992+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/1761894225 >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 0x7f0cf00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.993+0000 7f0ceffff640 1 --2- 172.21.15.1:0/1761894225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 0x7f0cf00d29e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80097a0 tx=0x7f0ce802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f57754ad04eb113 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.993+0000 7f0cee7fc640 1 -- 172.21.15.1:0/1761894225 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce802f050 con 0x7f0cf00b4ba0 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.993+0000 7f0cee7fc640 1 -- 172.21.15.1:0/1761894225 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce8031700 con 0x7f0cf00b4ba0 2024-03-24T10:35:39.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.993+0000 7f0cee7fc640 1 -- 172.21.15.1:0/1761894225 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031a90 con 0x7f0cf00b4ba0 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/1761894225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 msgr2=0x7f0cf00d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/1761894225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 0x7f0cf00d29e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80097a0 tx=0x7f0ce802b7a0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/1761894225 shutdown_connections 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/1761894225 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf00b4ba0 0x7f0cf00d29e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/1761894225 >> v2:172.21.15.3:3300/0 conn(0x7f0cf015e390 0x7f0cf00d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/1761894225 >> 172.21.15.1:0/1761894225 conn(0x7f0cf00d3ec0 msgr2=0x7f0cf00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/1761894225 shutdown_connections 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.994+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/1761894225 wait complete. 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.995+0000 7f0cf5e0b640 1 Processor -- start 2024-03-24T10:35:39.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.995+0000 7f0cf5e0b640 1 -- start start 2024-03-24T10:35:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf00b4ba0 0x7f0cf01562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 0x7f0cf009f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf00a1b30 con 0x7f0cf00d1000 2024-03-24T10:35:40.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf009f470 con 0x7f0cf00b4ba0 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cf5e0b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf009e860 con 0x7f0cf015e390 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50352/0 (socket says 172.21.15.1:50352) 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 -- 172.21.15.1:0/4292047030 learned_addr learned my addr 172.21.15.1:0/4292047030 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf00b4ba0 0x7f0cf01562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:40.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceffff640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 0x7f0cf009f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 -- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 msgr2=0x7f0cf009f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 0x7f0cf009f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 -- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf00b4ba0 msgr2=0x7f0cf01562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf00b4ba0 0x7f0cf01562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 -- 172.21.15.1:0/4292047030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce0009a70 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceffff640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 0x7f0cf009f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0ceeffd640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c730 tx=0x7f0ce400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.996+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4005ab0 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.997+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce4005c10 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.997+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce8009400 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.997+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4001d30 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.997+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf009ee20 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.998+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf009f620 con 0x7f0cf00d1000 2024-03-24T10:35:40.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.998+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0ce4007070 con 0x7f0cf00d1000 2024-03-24T10:35:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.998+0000 7f0cd7fff640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 0x7f0cdc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.998+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f0ce4089270 con 0x7f0cf00d1000 2024-03-24T10:35:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.999+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 0x7f0cdc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:39.999+0000 7f0cef7fe640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 0x7f0cdc06dd00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0007560 tx=0x7f0ce000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:40.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.003+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0cf009f620 con 0x7f0cf00d1000 2024-03-24T10:35:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.221+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cf015e770 con 0x7f0cf00d1000 2024-03-24T10:35:40.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.221+0000 7f0cd7fff640 1 -- 172.21.15.1:0/4292047030 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3851 (secure 0 0 0) 0x7f0cf015e770 con 0x7f0cf00d1000 2024-03-24T10:35:40.222 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:40.222 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/610749801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":610749801},{"type":"v1","addr":"172.21.15.3:6834","nonce":610749801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/319702132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":319702132},{"type":"v1","addr":"172.21.15.1:6835","nonce":319702132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2466806224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":2466806224},{"type":"v1","addr":"172.21.15.3:6835","nonce":2466806224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5877,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2118799247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2118799247},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118799247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-03-24T10:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 19 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 msgr2=0x7f0cdc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 0x7f0cdc06dd00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0007560 tx=0x7f0ce000b040 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 msgr2=0x7f0cf0156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c730 tx=0x7f0ce400cbf0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 shutdown_connections 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0cdc06b840 0x7f0cdc06dd00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0cf015e390 0x7f0cf009f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf00d1000 0x7f0cf0156cb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 --2- 172.21.15.1:0/4292047030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf00b4ba0 0x7f0cf01562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.223+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 >> 172.21.15.1:0/4292047030 conn(0x7f0cf00d3ec0 msgr2=0x7f0cf00b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.224+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 shutdown_connections 2024-03-24T10:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:40.224+0000 7f0cf5e0b640 1 -- 172.21.15.1:0/4292047030 wait complete. 2024-03-24T10:35:40.235 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T10:35:40.235 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T10:35:40.235 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T10:35:40.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:40.236 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T10:35:40.236 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T10:35:40.236 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T10:35:40.236 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:40.236 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T10:35:40.236 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T10:35:40.236 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T10:35:40.236 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:40.237 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T10:35:40.237 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T10:35:40.237 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T10:35:40.237 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:35:40.246 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T10:35:40.246 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T10:35:40.249 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T10:35:40.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T10:35:40.250 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T10:35:40.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T10:35:40.289 INFO:tasks.ceph.mds.c:Started 2024-03-24T10:35:40.289 DEBUG:teuthology.parallel:result is None 2024-03-24T10:35:40.293 INFO:tasks.ceph.mds.b:Started 2024-03-24T10:35:40.293 DEBUG:teuthology.parallel:result is None 2024-03-24T10:35:40.293 INFO:tasks.ceph.mds.d:Started 2024-03-24T10:35:40.294 DEBUG:teuthology.parallel:result is None 2024-03-24T10:35:40.363 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-24T10:35:40.380 INFO:tasks.ceph.mds.b.smithi003.stdout:starting mds.b at 2024-03-24T10:35:40.380 INFO:tasks.ceph.mds.d.smithi003.stdout:starting mds.d at 2024-03-24T10:35:41.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-24T10:35:46.246 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T10:35:46.246 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T10:35:46.250 INFO:tasks.ceph.mds.a:Started 2024-03-24T10:35:46.251 DEBUG:teuthology.parallel:result is None 2024-03-24T10:35:46.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T10:35:46.335 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-24T10:35:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.364+0000 7fd595cdf640 1 Processor -- start 2024-03-24T10:35:46.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 -- start start 2024-03-24T10:35:46.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd59015e390 0x7fd5900d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59015e770 con 0x7fd5900d1000 2024-03-24T10:35:46.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd5900b3190 con 0x7fd59015e390 2024-03-24T10:35:46.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd595cdf640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5900d4e40 con 0x7fd5900b4ba0 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd58effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd5900d1000 0x7fd5900cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51376/0 (socket says 172.21.15.1:51376) 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.365+0000 7fd58effd640 1 -- 172.21.15.1:0/2311389880 learned_addr learned my addr 172.21.15.1:0/2311389880 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58f7fe640 1 --2- 172.21.15.1:0/2311389880 >> v2:172.21.15.3:3300/0 conn(0x7fd59015e390 0x7fd5900d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58ffff640 1 --2- 172.21.15.1:0/2311389880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58e7fc640 1 -- 172.21.15.1:0/2311389880 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035051793 0 0) 0x7fd584001110 con 0x7fd5900d1000 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58e7fc640 1 -- 172.21.15.1:0/2311389880 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd578003520 con 0x7fd5900d1000 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58ffff640 1 -- 172.21.15.1:0/2311389880 >> v2:172.21.15.3:3300/0 conn(0x7fd59015e390 msgr2=0x7fd5900d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58ffff640 1 --2- 172.21.15.1:0/2311389880 >> v2:172.21.15.3:3300/0 conn(0x7fd59015e390 0x7fd5900d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58ffff640 1 -- 172.21.15.1:0/2311389880 >> v1:172.21.15.1:6789/0 conn(0x7fd5900d1000 legacy=0x7fd5900cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:46.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.366+0000 7fd58ffff640 1 -- 172.21.15.1:0/2311389880 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900d1e40 con 0x7fd5900b4ba0 2024-03-24T10:35:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd58ffff640 1 --2- 172.21.15.1:0/2311389880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900d29e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd58802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0bd20275e234afe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd58e7fc640 1 -- 172.21.15.1:0/2311389880 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58802cb30 con 0x7fd5900b4ba0 2024-03-24T10:35:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd58e7fc640 1 -- 172.21.15.1:0/2311389880 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58802cc90 con 0x7fd5900b4ba0 2024-03-24T10:35:46.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd58e7fc640 1 -- 172.21.15.1:0/2311389880 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588034440 con 0x7fd5900b4ba0 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd595cdf640 1 -- 172.21.15.1:0/2311389880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 msgr2=0x7fd5900d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.367+0000 7fd595cdf640 1 --2- 172.21.15.1:0/2311389880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900d29e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd58802bd80 comp rx=0 tx=0).stop 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 -- 172.21.15.1:0/2311389880 shutdown_connections 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 --2- 172.21.15.1:0/2311389880 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900d29e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 --2- 172.21.15.1:0/2311389880 >> v2:172.21.15.3:3300/0 conn(0x7fd59015e390 0x7fd5900d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 -- 172.21.15.1:0/2311389880 >> 172.21.15.1:0/2311389880 conn(0x7fd5900d3ec0 msgr2=0x7fd5900d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 -- 172.21.15.1:0/2311389880 shutdown_connections 2024-03-24T10:35:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.368+0000 7fd595cdf640 1 -- 172.21.15.1:0/2311389880 wait complete. 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.369+0000 7fd595cdf640 1 Processor -- start 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.369+0000 7fd595cdf640 1 -- start start 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5900d1000 0x7fd59014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd5900b82a0 con 0x7fd59015e390 2024-03-24T10:35:46.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd5900ba060 con 0x7fd5900d1000 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd595cdf640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5900b9a10 con 0x7fd5900b4ba0 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53208/0 (socket says 172.21.15.1:53208) 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 -- 172.21.15.1:0/663908234 learned_addr learned my addr 172.21.15.1:0/663908234 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58f7fe640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58effd640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5900d1000 0x7fd59014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 -- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 msgr2=0x7fd5900b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 -- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5900d1000 msgr2=0x7fd59014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5900d1000 0x7fd59014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 -- 172.21.15.1:0/663908234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd588009a70 con 0x7fd59015e390 2024-03-24T10:35:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58f7fe640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.370+0000 7fd58ffff640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd588004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.371+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588007000 con 0x7fd59015e390 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.371+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd588040450 con 0x7fd59015e390 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.371+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5900b7620 con 0x7fd59015e390 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.371+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5880407e0 con 0x7fd59015e390 2024-03-24T10:35:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.371+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5900b7be0 con 0x7fd59015e390 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.372+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5900b83e0 con 0x7fd59015e390 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.372+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd588040940 con 0x7fd59015e390 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.373+0000 7fd577fff640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.373+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4943+0+0 (secure 0 0 0) 0x7fd5880acb20 con 0x7fd59015e390 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.373+0000 7fd58f7fe640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:46.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.373+0000 7fd58f7fe640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 0x7fd57c06dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd580009af0 tx=0x7fd580008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:46.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.377+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd5900b83e0 con 0x7fd59015e390 2024-03-24T10:35:46.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:46.570+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd5900b83e0 con 0x7fd59015e390 2024-03-24T10:35:47.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.593+0000 7fd577fff640 1 -- 172.21.15.1:0/663908234 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v46) ==== 97+0+0 (secure 0 0 0) 0x7fd5900b83e0 con 0x7fd59015e390 2024-03-24T10:35:47.593 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-24T10:35:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 msgr2=0x7fd57c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 0x7fd57c06dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd580009af0 tx=0x7fd580008040 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 msgr2=0x7fd590152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd588004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 shutdown_connections 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd59015e390 0x7fd590152990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5900d1000 0x7fd59014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 --2- 172.21.15.1:0/663908234 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5900b4ba0 0x7fd5900b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.596+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 >> 172.21.15.1:0/663908234 conn(0x7fd5900d3ec0 msgr2=0x7fd5900d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.597+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 shutdown_connections 2024-03-24T10:35:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.597+0000 7fd595cdf640 1 -- 172.21.15.1:0/663908234 wait complete. 2024-03-24T10:35:47.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T10:35:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.718+0000 7f230efc9640 1 Processor -- start 2024-03-24T10:35:47.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.718+0000 7f230efc9640 1 -- start start 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230efc9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2308163660 0x7f23080726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230efc9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23081458b0 con 0x7f23080b42a0 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230efc9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2308071c80 con 0x7f2308147b80 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230efc9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f23081486b0 con 0x7f2308163660 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230cd65640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2308163660 0x7f23080726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230d566640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f23080b42a0 0x7f230806b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51382/0 (socket says 172.21.15.1:51382) 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230d566640 1 -- 172.21.15.1:0/2744439142 learned_addr learned my addr 172.21.15.1:0/2744439142 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.719+0000 7f230cd65640 1 --2- 172.21.15.1:0/2744439142 >> v2:172.21.15.1:3300/0 conn(0x7f2308163660 0x7f23080726d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53224/0 (socket says 172.21.15.1:53224) 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.720+0000 7f230cd65640 1 -- 172.21.15.1:0/2744439142 >> v1:172.21.15.3:6789/0 conn(0x7f2308147b80 legacy=0x7f23080b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.720+0000 7f230cd65640 1 -- 172.21.15.1:0/2744439142 >> v1:172.21.15.1:6789/0 conn(0x7f23080b42a0 legacy=0x7f230806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.720+0000 7f230cd65640 1 -- 172.21.15.1:0/2744439142 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2308148990 con 0x7f2308163660 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.720+0000 7f230cd65640 1 --2- 172.21.15.1:0/2744439142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308163660 0x7f23080726d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0097d0 tx=0x7f22fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ca319351d12203 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.720+0000 7f23077fe640 1 -- 172.21.15.1:0/2744439142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc030610 con 0x7f2308163660 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f23077fe640 1 -- 172.21.15.1:0/2744439142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22fc030770 con 0x7f2308163660 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f23077fe640 1 -- 172.21.15.1:0/2744439142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc030b00 con 0x7f2308163660 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 -- 172.21.15.1:0/2744439142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308163660 msgr2=0x7f23080726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 --2- 172.21.15.1:0/2744439142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308163660 0x7f23080726d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0097d0 tx=0x7f22fc02b960 comp rx=0 tx=0).stop 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 -- 172.21.15.1:0/2744439142 shutdown_connections 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 --2- 172.21.15.1:0/2744439142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308163660 0x7f23080726d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 -- 172.21.15.1:0/2744439142 >> 172.21.15.1:0/2744439142 conn(0x7f23080b2890 msgr2=0x7f23080b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 -- 172.21.15.1:0/2744439142 shutdown_connections 2024-03-24T10:35:47.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.721+0000 7f230efc9640 1 -- 172.21.15.1:0/2744439142 wait complete. 2024-03-24T10:35:47.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.722+0000 7f230efc9640 1 Processor -- start 2024-03-24T10:35:47.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.722+0000 7f230efc9640 1 -- start start 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23080b42a0 0x7f2308137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2308163660 0x7f2308144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2308139160 con 0x7f2308147b80 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f230813aeb0 con 0x7f2308163660 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f230efc9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f2308138550 con 0x7f23080b42a0 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f2307fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f2307fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53236/0 (socket says 172.21.15.1:53236) 2024-03-24T10:35:47.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.723+0000 7f2307fff640 1 -- 172.21.15.1:0/1292653339 learned_addr learned my addr 172.21.15.1:0/1292653339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f230d566640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2308163660 0x7f2308144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f230cd65640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23080b42a0 0x7f2308137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 -- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23080b42a0 msgr2=0x7f2308137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23080b42a0 0x7f2308137eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 -- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2308163660 msgr2=0x7f2308144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2308163660 0x7f2308144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:47.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 -- 172.21.15.1:0/1292653339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc009400 con 0x7f2308147b80 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f2307fff640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c400 tx=0x7f22f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f40052d0 con 0x7f2308147b80 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f4005450 con 0x7f2308147b80 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4007260 con 0x7f2308147b80 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2308138830 con 0x7f2308147b80 2024-03-24T10:35:47.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.724+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2308138df0 con 0x7f2308147b80 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.725+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308139310 con 0x7f2308147b80 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.726+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22f40073c0 con 0x7f2308147b80 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.726+0000 7f23057fa640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 0x7f22ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.726+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4857+0+0 (secure 0 0 0) 0x7f22f40892b0 con 0x7f2308147b80 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.726+0000 7f230cd65640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 0x7f22ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:47.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.727+0000 7f230cd65640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 0x7f22ec06dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:47.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.731+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2308139310 con 0x7f2308147b80 2024-03-24T10:35:47.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.978+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f230804f4e0 con 0x7f2308147b80 2024-03-24T10:35:47.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.979+0000 7f23057fa640 1 -- 172.21.15.1:0/1292653339 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5298 (secure 0 0 0) 0x7f22f4054020 con 0x7f2308147b80 2024-03-24T10:35:47.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmJeY4FBAA68H0WQ/mqjsr0lZfBIvuIg==", 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm2p+QIBAAokf+P5vhhBqszwKd+fn11w==", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-24T10:35:47.981 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmjiKEGxAA9ZjhlLO78jdVtdk4FZILhA==", 2024-03-24T10:35:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.982 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm+dgUKBAAG62jv3f5tbgZc3yHG/BiFQ==", 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:35:47.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBcAQBmcF+vEhAA6d6m64/OD24zfbP1RjYR1w==", 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-24T10:35:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBfAQBmOi2fERAAJ3x4Y4d96THYK1bBkTqqPg==", 2024-03-24T10:35:47.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBiAQBmhsQJEBAAGwQdvTTDG3liCqR+8Y1++w==", 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBlAQBmzW42DhAAopMY+PpT0hSNL1Y/h0rOmQ==", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-24T10:35:47.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBrAQBmqZxLFRAAoXQjHMDUYKqwpqKunxNuzg==", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBuAQBmIQoSFBAAgw2p6P1rqsdn94zL8BCmDg==", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBxAQBmnxrVEhAAthwnDMkD5AyCr0gCPvE3ZQ==", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQB0AQBm8UNjERAARFrNk44p63MiF0L0YVsAqw==", 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-24T10:35:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-24T10:35:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw==", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmO0d+NRAAhw9H+eNh7wKil/BAbqbQSQ==", 2024-03-24T10:35:47.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm1Z8YMhAArYu32NrH0OcQYcX13UCsnQ==", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBm9xNHOxAAAxXBCXt6EiKWgso4flywRQ==", 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-24T10:35:47.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBZAQBmpZqfMhAAB5KUBr9dFz54a442ZCMWIg==", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.995 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmBLzREBAA9Hph9Xljf0eGaV92TSLlfg==", 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-24T10:35:47.996 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQBaAQBmHPHHDBAAIuuLFGLFsRogrQjUx44H7A==", 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-24T10:35:47.997 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 msgr2=0x7f22ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 0x7f22ec06dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f22fc036000 tx=0x7f22fc03b040 comp rx=0 tx=0).stop 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 msgr2=0x7f230813a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c400 tx=0x7f22f400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 shutdown_connections 2024-03-24T10:35:48.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f22ec06b810 0x7f22ec06dcd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2308163660 0x7f2308144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2308147b80 0x7f230813a970 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 --2- 172.21.15.1:0/1292653339 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23080b42a0 0x7f2308137eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 >> 172.21.15.1:0/1292653339 conn(0x7f23080b2890 msgr2=0x7f230815c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 shutdown_connections 2024-03-24T10:35:48.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:47.981+0000 7f230efc9640 1 -- 172.21.15.1:0/1292653339 wait complete. 2024-03-24T10:35:48.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:48.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.107+0000 7fb81fde9640 1 Processor -- start 2024-03-24T10:35:48.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.107+0000 7fb81fde9640 1 -- start start 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81fde9640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb81815e390 0x7fb8180d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81fde9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8180d1000 0x7fb8180d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81fde9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb81815e770 con 0x7fb8180b3320 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81fde9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb8180b3190 con 0x7fb8180d1000 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81fde9640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb8180d4e40 con 0x7fb81815e390 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81d384640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8180d1000 0x7fb8180d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81d384640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8180d1000 0x7fb8180d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53246/0 (socket says 172.21.15.1:53246) 2024-03-24T10:35:48.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.108+0000 7fb81d384640 1 -- 172.21.15.1:0/1617176839 learned_addr learned my addr 172.21.15.1:0/1617176839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81db85640 1 --2- 172.21.15.1:0/1617176839 >> v2:172.21.15.3:3300/0 conn(0x7fb81815e390 0x7fb8180d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81d384640 1 -- 172.21.15.1:0/1617176839 >> v2:172.21.15.3:3300/0 conn(0x7fb81815e390 msgr2=0x7fb8180d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81d384640 1 --2- 172.21.15.1:0/1617176839 >> v2:172.21.15.3:3300/0 conn(0x7fb81815e390 0x7fb8180d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81d384640 1 -- 172.21.15.1:0/1617176839 >> v1:172.21.15.3:6789/0 conn(0x7fb8180b3320 legacy=0x7fb8180cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81d384640 1 -- 172.21.15.1:0/1617176839 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8180d1e40 con 0x7fb8180d1000 2024-03-24T10:35:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.109+0000 7fb81d384640 1 --2- 172.21.15.1:0/1617176839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8180d1000 0x7fb8180d29e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb8180d3040 tx=0x7fb80c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dfca83e5536fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81cb83640 1 -- 172.21.15.1:0/1617176839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c0317e0 con 0x7fb8180d1000 2024-03-24T10:35:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81cb83640 1 -- 172.21.15.1:0/1617176839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80c031940 con 0x7fb8180d1000 2024-03-24T10:35:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81cb83640 1 -- 172.21.15.1:0/1617176839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c031cd0 con 0x7fb8180d1000 2024-03-24T10:35:48.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 -- 172.21.15.1:0/1617176839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8180d1000 msgr2=0x7fb8180d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1617176839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8180d1000 0x7fb8180d29e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb8180d3040 tx=0x7fb80c02b880 comp rx=0 tx=0).stop 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 -- 172.21.15.1:0/1617176839 shutdown_connections 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1617176839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8180d1000 0x7fb8180d29e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1617176839 >> v2:172.21.15.3:3300/0 conn(0x7fb81815e390 0x7fb8180d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 -- 172.21.15.1:0/1617176839 >> 172.21.15.1:0/1617176839 conn(0x7fb8180d3ec0 msgr2=0x7fb8180d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 -- 172.21.15.1:0/1617176839 shutdown_connections 2024-03-24T10:35:48.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.110+0000 7fb81fde9640 1 -- 172.21.15.1:0/1617176839 wait complete. 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.111+0000 7fb81fde9640 1 Processor -- start 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.111+0000 7fb81fde9640 1 -- start start 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 0x7fb818151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 0x7fb8180cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb81809dff0 con 0x7fb81815e390 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb8180cb130 con 0x7fb8180d1000 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81fde9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb8180cb410 con 0x7fb8180b3320 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81e386640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81e386640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53250/0 (socket says 172.21.15.1:53250) 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.112+0000 7fb81e386640 1 -- 172.21.15.1:0/1933735726 learned_addr learned my addr 172.21.15.1:0/1933735726 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81db85640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 0x7fb818151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81d384640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 0x7fb8180cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 -- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 msgr2=0x7fb818151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 0x7fb818151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 -- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 msgr2=0x7fb8180cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 0x7fb8180cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 -- 172.21.15.1:0/1933735726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c0094d0 con 0x7fb81815e390 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81db85640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 0x7fb818151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81d384640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 0x7fb8180cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81e386640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb810001eb0 tx=0x7fb81000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100052d0 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb810005430 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8180cb6f0 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb810003810 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.113+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8180ce1a0 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.114+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8180cbc10 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.114+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb810003970 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.115+0000 7fb8167fc640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 0x7fb80406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.115+0000 7fb81db85640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 0x7fb80406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.115+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4857+0+0 (secure 0 0 0) 0x7fb8100892b0 con 0x7fb81815e390 2024-03-24T10:35:48.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.115+0000 7fb81db85640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 0x7fb80406dcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb808009f50 tx=0x7fb80800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.119+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb8180cbc10 con 0x7fb81815e390 2024-03-24T10:35:48.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.302+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb81815e770 con 0x7fb81815e390 2024-03-24T10:35:48.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.303+0000 7fb8167fc640 1 -- 172.21.15.1:0/1933735726 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+12759 (secure 0 0 0) 0x7fb81815e770 con 0x7fb81815e390 2024-03-24T10:35:48.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:48.303 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":46,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:47.592410+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 msgr2=0x7fb80406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 0x7fb80406dcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb808009f50 tx=0x7fb80800b040 comp rx=0 tx=0).stop 2024-03-24T10:35:48.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 msgr2=0x7fb81806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb810001eb0 tx=0x7fb81000c890 comp rx=0 tx=0).stop 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 shutdown_connections 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb80406b810 0x7fb80406dcd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb81815e390 0x7fb81806c730 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8180d1000 0x7fb8180cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 --2- 172.21.15.1:0/1933735726 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb8180b3320 0x7fb818151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 >> 172.21.15.1:0/1933735726 conn(0x7fb8180d3ec0 msgr2=0x7fb8180b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 shutdown_connections 2024-03-24T10:35:48.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.305+0000 7fb81fde9640 1 -- 172.21.15.1:0/1933735726 wait complete. 2024-03-24T10:35:48.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:35:48.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:35:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.421+0000 7fe8dd483640 1 Processor -- start 2024-03-24T10:35:48.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.421+0000 7fe8dd483640 1 -- start start 2024-03-24T10:35:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8dd483640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8d806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8dd483640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d80cc2b0 con 0x7fe8d80ccd00 2024-03-24T10:35:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8dd483640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d80cc870 con 0x7fe8d815e2c0 2024-03-24T10:35:48.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8dd483640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d80d0d50 con 0x7fe8d806fe90 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe8d80ccd00 0x7fe8d80cd190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51384/0 (socket says 172.21.15.1:51384) 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d77fe640 1 -- 172.21.15.1:0/3056938191 learned_addr learned my addr 172.21.15.1:0/3056938191 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8ce7fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8d806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523908320 0 0) 0x7fe8d80cc2b0 con 0x7fe8d80ccd00 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8c4003520 con 0x7fe8d80ccd00 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1085945858 0 0) 0x7fe8c4003520 con 0x7fe8d80ccd00 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 msgr2=0x7fe8d806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8d806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 >> v1:172.21.15.3:6789/0 conn(0x7fe8d815e2c0 legacy=0x7fe8d806f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d80d1030 con 0x7fe8d80ccd00 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fe8d00019f0 con 0x7fe8d80ccd00 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.422+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 >> v1:172.21.15.1:6789/0 conn(0x7fe8d80ccd00 legacy=0x7fe8d80cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8c40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8c4003ee0 0x7fe8c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8c4004d00 0x7fe8c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d80d0d50 con 0x7fe8c4004d00 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c400ba90 con 0x7fe8c4003ee0 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c400bc50 con 0x7fe8d806fe90 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8c4004d00 0x7fe8c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8ce7fc640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8c4003ee0 0x7fe8c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d6ffd640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8c40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 -- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 msgr2=0x7fe8c40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d806fe90 0x7fe8c40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 -- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8c4003ee0 msgr2=0x7fe8c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8c4003ee0 0x7fe8c40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.423+0000 7fe8d77fe640 1 -- 172.21.15.1:0/3056938191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c4003520 con 0x7fe8c4004d00 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8c4004d00 0x7fe8c4005180 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000a830 tx=0x7fe8c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8aad7e62ddfd4af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d002cea0 con 0x7fe8c4004d00 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d002f070 con 0x7fe8c4004d00 2024-03-24T10:35:48.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8d67fc640 1 -- 172.21.15.1:0/3056938191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00348d0 con 0x7fe8c4004d00 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 -- 172.21.15.1:0/3056938191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8c4004d00 msgr2=0x7fe8c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 --2- 172.21.15.1:0/3056938191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8c4004d00 0x7fe8c4005180 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000a830 tx=0x7fe8c40057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8d77fe640 1 -- 172.21.15.1:0/3056938191 reap_dead start 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 -- 172.21.15.1:0/3056938191 shutdown_connections 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 -- 172.21.15.1:0/3056938191 >> 172.21.15.1:0/3056938191 conn(0x7fe8d80d1c60 msgr2=0x7fe8d80d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 -- 172.21.15.1:0/3056938191 shutdown_connections 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.424+0000 7fe8dd483640 1 -- 172.21.15.1:0/3056938191 wait complete. 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 Processor -- start 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 -- start start 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d80b41f0 0x7fe8d80b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8d81f77a0 0x7fe8d814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d81f7380 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d80ce260 con 0x7fe8d81f77a0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.426+0000 7fe8dd483640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d81f6d50 con 0x7fe8d80b41f0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53266/0 (socket says 172.21.15.1:53266) 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 -- 172.21.15.1:0/2358762727 learned_addr learned my addr 172.21.15.1:0/2358762727 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d6ffd640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d80b41f0 0x7fe8d80b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8ce7fc640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8d81f77a0 0x7fe8d814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 -- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d80b41f0 msgr2=0x7fe8d80b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d80b41f0 0x7fe8d80b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 -- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8d81f77a0 msgr2=0x7fe8d814ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8d81f77a0 0x7fe8d814ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 -- 172.21.15.1:0/2358762727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d814cf80 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8d77fe640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000a950 tx=0x7fe8d0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0034840 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d00349a0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d80cb840 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0034d50 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.427+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d81f6fa0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.428+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d81f74c0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.429+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe8d0031030 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.429+0000 7fe8cffff640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 0x7fe8b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.429+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4857+0+0 (secure 0 0 0) 0x7fe8d00abf50 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.429+0000 7fe8d6ffd640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 0x7fe8b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.430+0000 7fe8d6ffd640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 0x7fe8b806dd00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8003700 tx=0x7fe8c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.433+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe8d81f74c0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-24T10:35:48.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.614+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8d81f74c0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.614+0000 7fe8cffff640 1 -- 172.21.15.1:0/2358762727 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) ==== 93+0+10 (secure 0 0 0) 0x7fe8d81f74c0 con 0x7fe8d80cd8a0 2024-03-24T10:35:48.617 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:35:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.616+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 msgr2=0x7fe8b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.616+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 0x7fe8b806dd00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8003700 tx=0x7fe8c8009040 comp rx=0 tx=0).stop 2024-03-24T10:35:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.616+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 msgr2=0x7fe8d80cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.616+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000a950 tx=0x7fe8d0001d80 comp rx=0 tx=0).stop 2024-03-24T10:35:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 shutdown_connections 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe8b806b840 0x7fe8b806dd00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d80cd8a0 0x7fe8d80cdd20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8d81f77a0 0x7fe8d814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 --2- 172.21.15.1:0/2358762727 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe8d80b41f0 0x7fe8d80b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 >> 172.21.15.1:0/2358762727 conn(0x7fe8d80d1c60 msgr2=0x7fe8d80d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 shutdown_connections 2024-03-24T10:35:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.617+0000 7fe8dd483640 1 -- 172.21.15.1:0/2358762727 wait complete. 2024-03-24T10:35:48.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T10:35:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 Processor -- start 2024-03-24T10:35:48.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 -- start start 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd3300b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3300b37c0 con 0x7fd33006cc00 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd330071120 con 0x7fd330071ea0 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.731+0000 7fd3360ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd3300b34c0 con 0x7fd330071ac0 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd33006cc00 0x7fd3300b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51386/0 (socket says 172.21.15.1:51386) 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32effd640 1 -- 172.21.15.1:0/1799242369 learned_addr learned my addr 172.21.15.1:0/1799242369 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653542816 0 0) 0x7fd3300b37c0 con 0x7fd33006cc00 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd318005730 con 0x7fd33006cc00 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32f7fe640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd3300b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2522408756 0 0) 0x7fd318005730 con 0x7fd33006cc00 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 msgr2=0x7fd3300b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd3300b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 >> v1:172.21.15.3:6789/0 conn(0x7fd330071ea0 legacy=0x7fd33015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33006e660 con 0x7fd33006cc00 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fd324001670 con 0x7fd33006cc00 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 >> v1:172.21.15.1:6789/0 conn(0x7fd33006cc00 legacy=0x7fd3300b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd318005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3180060f0 0x7fd3180069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318006f10 0x7fd318007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3300b34c0 con 0x7fd318006f10 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd31800dca0 con 0x7fd3180060f0 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd31800de60 con 0x7fd330071ac0 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32effd640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3180060f0 0x7fd3180069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32ffff640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318006f10 0x7fd318007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.732+0000 7fd32f7fe640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd318005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 -- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 msgr2=0x7fd318005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd330071ac0 0x7fd318005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 -- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3180060f0 msgr2=0x7fd3180069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:48.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3180060f0 0x7fd3180069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 -- 172.21.15.1:0/1799242369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd318005730 con 0x7fd318006f10 2024-03-24T10:35:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.733+0000 7fd32ffff640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318006f10 0x7fd318007390 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd328001f60 tx=0x7fd318007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a804b9defc7d90ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32802d6f0 con 0x7fd318006f10 2024-03-24T10:35:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd32802d870 con 0x7fd318006f10 2024-03-24T10:35:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd32e7fc640 1 -- 172.21.15.1:0/1799242369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32802dc20 con 0x7fd318006f10 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 -- 172.21.15.1:0/1799242369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318006f10 msgr2=0x7fd318007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 --2- 172.21.15.1:0/1799242369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318006f10 0x7fd318007390 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd328001f60 tx=0x7fd318007a00 comp rx=0 tx=0).stop 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd32ffff640 1 -- 172.21.15.1:0/1799242369 reap_dead start 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 -- 172.21.15.1:0/1799242369 shutdown_connections 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 -- 172.21.15.1:0/1799242369 >> 172.21.15.1:0/1799242369 conn(0x7fd3300b1710 msgr2=0x7fd3300b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 -- 172.21.15.1:0/1799242369 shutdown_connections 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.734+0000 7fd3360ae640 1 -- 172.21.15.1:0/1799242369 wait complete. 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.735+0000 7fd3360ae640 1 Processor -- start 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.735+0000 7fd3360ae640 1 -- start start 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd330151650 0x7fd330153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd33015a830 0x7fd3301539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd330153180 con 0x7fd330153f10 2024-03-24T10:35:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd33015ada0 con 0x7fd330151650 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd3360ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd33015a4e0 con 0x7fd33015a830 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53282/0 (socket says 172.21.15.1:53282) 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32ffff640 1 -- 172.21.15.1:0/2913545516 learned_addr learned my addr 172.21.15.1:0/2913545516 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32effd640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd33015a830 0x7fd3301539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32f7fe640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd330151650 0x7fd330153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.736+0000 7fd32ffff640 1 -- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd33015a830 msgr2=0x7fd3301539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd32ffff640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd33015a830 0x7fd3301539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd32ffff640 1 -- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd330151650 msgr2=0x7fd330153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd32ffff640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd330151650 0x7fd330153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd32ffff640 1 -- 172.21.15.1:0/2913545516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd330159f20 con 0x7fd330153f10 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd32ffff640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd328037040 tx=0x7fd3280015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32802dea0 con 0x7fd330153f10 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd328003b70 con 0x7fd330153f10 2024-03-24T10:35:48.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33015a200 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd328005070 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.737+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd330151a30 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.738+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3300cd040 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.738+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3280051d0 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.739+0000 7fd317fff640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 0x7fd31c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.739+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4857+0+0 (secure 0 0 0) 0x7fd328031070 con 0x7fd330153f10 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.739+0000 7fd32f7fe640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 0x7fd31c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:48.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.739+0000 7fd32f7fe640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 0x7fd31c06dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd320009b00 tx=0x7fd320008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:48.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.743+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd328076cc0 con 0x7fd330153f10 2024-03-24T10:35:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.924+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd33004f4e0 con 0x7fd330153f10 2024-03-24T10:35:48.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.924+0000 7fd317fff640 1 -- 172.21.15.1:0/2913545516 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) ==== 84+0+65 (secure 0 0 0) 0x7fd3280011c0 con 0x7fd330153f10 2024-03-24T10:35:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:48.926 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-24T10:35:48.926 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-24T10:35:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-24T10:35:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-24T10:35:48.927 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:35:48.927 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 msgr2=0x7fd31c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 0x7fd31c06dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd320009b00 tx=0x7fd320008040 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 msgr2=0x7fd33015c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd328037040 tx=0x7fd3280015b0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 shutdown_connections 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd31c06b810 0x7fd31c06dcd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd330153f10 0x7fd33015c310 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd33015a830 0x7fd3301539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 --2- 172.21.15.1:0/2913545516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd330151650 0x7fd330153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 >> 172.21.15.1:0/2913545516 conn(0x7fd3300b1710 msgr2=0x7fd3300b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 shutdown_connections 2024-03-24T10:35:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:48.926+0000 7fd3360ae640 1 -- 172.21.15.1:0/2913545516 wait complete. 2024-03-24T10:35:48.937 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T10:35:48.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T10:35:49.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.042+0000 7f489538a640 1 Processor -- start 2024-03-24T10:35:49.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.042+0000 7f489538a640 1 -- start start 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f489538a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f48900d0690 0x7f4890165bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f489538a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48900b45c0 con 0x7f489015e860 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f489538a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f489015ed00 con 0x7f48900cbf90 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f489538a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f489006c930 con 0x7f48900d0690 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f4887fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f489015e860 0x7f48900cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51390/0 (socket says 172.21.15.1:51390) 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f4887fff640 1 -- 172.21.15.1:0/406445581 learned_addr learned my addr 172.21.15.1:0/406445581 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f48900d0690 0x7f4890165bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524045063 0 0) 0x7f48900b45c0 con 0x7f489015e860 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f487c003520 con 0x7f489015e860 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 182235509 0 0) 0x7f487c003520 con 0x7f489015e860 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f48900d0690 msgr2=0x7f4890165bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f48900d0690 0x7f4890165bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 >> v1:172.21.15.3:6789/0 conn(0x7f48900cbf90 legacy=0x7f489015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489015efe0 con 0x7f489015e860 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f4878001670 con 0x7f489015e860 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 >> v1:172.21.15.1:6789/0 conn(0x7f489015e860 legacy=0x7f48900cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0039a0 0x7f487c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f487c0047c0 0x7f487c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.043+0000 7f488e7fc640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0051d0 0x7f487c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f489006c930 con 0x7f487c0051d0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f487c00c0d0 con 0x7f487c0047c0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f487c00c290 con 0x7f487c0039a0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f48900d0690 0x7f4890165bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0051d0 0x7f487c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488effd640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0039a0 0x7f487c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f4887fff640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f487c0047c0 0x7f487c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 -- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0039a0 msgr2=0x7f487c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0039a0 0x7f487c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 -- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f487c0047c0 msgr2=0x7f487c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f487c0047c0 0x7f487c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f488f7fe640 1 -- 172.21.15.1:0/406445581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c003520 con 0x7f487c0051d0 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.044+0000 7f4887fff640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f487c0047c0 0x7f487c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f488f7fe640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0051d0 0x7f487c00b980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4888009f00 tx=0x7f487c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518b856ef79db954 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488802cc30 con 0x7f487c0051d0 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488802cd90 con 0x7f487c0051d0 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f488e7fc640 1 -- 172.21.15.1:0/406445581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888003bb0 con 0x7f487c0051d0 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f489538a640 1 -- 172.21.15.1:0/406445581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0051d0 msgr2=0x7f487c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f489538a640 1 --2- 172.21.15.1:0/406445581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0051d0 0x7f487c00b980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4888009f00 tx=0x7f487c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.045+0000 7f488f7fe640 1 -- 172.21.15.1:0/406445581 reap_dead start 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.046+0000 7f489538a640 1 -- 172.21.15.1:0/406445581 shutdown_connections 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.046+0000 7f489538a640 1 -- 172.21.15.1:0/406445581 >> 172.21.15.1:0/406445581 conn(0x7f489006b4f0 msgr2=0x7f489006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.046+0000 7f489538a640 1 -- 172.21.15.1:0/406445581 shutdown_connections 2024-03-24T10:35:49.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.046+0000 7f489538a640 1 -- 172.21.15.1:0/406445581 wait complete. 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.047+0000 7f489538a640 1 Processor -- start 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.047+0000 7f489538a640 1 -- start start 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0051d0 0x7f489015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b2020 0x7f48900cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f48900c9b80 con 0x7f48901493e0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f48900c9e60 con 0x7f48900b2020 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f48900ca140 con 0x7f487c0051d0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53300/0 (socket says 172.21.15.1:53300) 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 -- 172.21.15.1:0/3500826249 learned_addr learned my addr 172.21.15.1:0/3500826249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488effd640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0051d0 0x7f489015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f4887fff640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b2020 0x7f48900cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 -- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0051d0 msgr2=0x7f489015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0051d0 0x7f489015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 -- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b2020 msgr2=0x7f48900cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b2020 0x7f48900cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 -- 172.21.15.1:0/3500826249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48900ca420 con 0x7f48901493e0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f488f7fe640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f488802c6a0 tx=0x7f4888004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880045c0 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4888004720 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48900ca700 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880376a0 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.048+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4890149890 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.049+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489004f470 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.050+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4888037800 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.050+0000 7f48877fe640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.050+0000 7f488effd640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.051+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4857+0+0 (secure 0 0 0) 0x7f48880ab9f0 con 0x7f48901493e0 2024-03-24T10:35:49.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.051+0000 7f488effd640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 0x7f487006dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f489015de50 tx=0x7f4880009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.055+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f48880768a0 con 0x7f48901493e0 2024-03-24T10:35:49.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.237+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f48900b3b40 con 0x7f48901493e0 2024-03-24T10:35:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.607+0000 7f48877fe640 1 -- 172.21.15.1:0/3500826249 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v47) ==== 138+0+0 (secure 0 0 0) 0x7f48880460b0 con 0x7f48901493e0 2024-03-24T10:35:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-24T10:35:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 msgr2=0x7f487006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 0x7f487006dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f489015de50 tx=0x7f4880009040 comp rx=0 tx=0).stop 2024-03-24T10:35:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 msgr2=0x7f4890149ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f488802c6a0 tx=0x7f4888004ab0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 shutdown_connections 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48901493e0 0x7f4890149ff0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b2020 0x7f48900cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 --2- 172.21.15.1:0/3500826249 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f487c0051d0 0x7f489015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 >> 172.21.15.1:0/3500826249 conn(0x7f489006b4f0 msgr2=0x7f4890159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.610+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 shutdown_connections 2024-03-24T10:35:49.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.611+0000 7f489538a640 1 -- 172.21.15.1:0/3500826249 wait complete. 2024-03-24T10:35:49.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T10:35:49.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.736+0000 7f1e4754f640 1 Processor -- start 2024-03-24T10:35:49.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.736+0000 7f1e4754f640 1 -- start start 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e4754f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1e401615f0 0x7f1e400d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e4754f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e400b4e40 0x7f1e40161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e4754f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e400c7a40 con 0x7f1e401619d0 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e4754f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1e400cfd50 con 0x7f1e400b4e40 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e4754f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f1e400b55b0 con 0x7f1e401615f0 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e44aea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e400b4e40 0x7f1e40161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e44aea640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e400b4e40 0x7f1e40161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53302/0 (socket says 172.21.15.1:53302) 2024-03-24T10:35:49.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.737+0000 7f1e44aea640 1 -- 172.21.15.1:0/2615782318 learned_addr learned my addr 172.21.15.1:0/2615782318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:49.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e452eb640 1 --2- 172.21.15.1:0/2615782318 >> v2:172.21.15.3:3300/0 conn(0x7f1e401615f0 0x7f1e400d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e44aea640 1 -- 172.21.15.1:0/2615782318 >> v2:172.21.15.3:3300/0 conn(0x7f1e401615f0 msgr2=0x7f1e400d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:49.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e44aea640 1 --2- 172.21.15.1:0/2615782318 >> v2:172.21.15.3:3300/0 conn(0x7f1e401615f0 0x7f1e400d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e44aea640 1 -- 172.21.15.1:0/2615782318 >> v1:172.21.15.3:6789/0 conn(0x7f1e401619d0 legacy=0x7f1e400d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:49.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e44aea640 1 -- 172.21.15.1:0/2615782318 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e400b52d0 con 0x7f1e400b4e40 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.738+0000 7f1e44aea640 1 --2- 172.21.15.1:0/2615782318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e400b4e40 0x7f1e40161f00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1e400b69f0 tx=0x7f1e3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf2c5c879641afc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e3ffff640 1 -- 172.21.15.1:0/2615782318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34031870 con 0x7f1e400b4e40 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e3ffff640 1 -- 172.21.15.1:0/2615782318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e340319d0 con 0x7f1e400b4e40 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e3ffff640 1 -- 172.21.15.1:0/2615782318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34031d60 con 0x7f1e400b4e40 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 -- 172.21.15.1:0/2615782318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e400b4e40 msgr2=0x7f1e40161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 --2- 172.21.15.1:0/2615782318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e400b4e40 0x7f1e40161f00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1e400b69f0 tx=0x7f1e3402b910 comp rx=0 tx=0).stop 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 -- 172.21.15.1:0/2615782318 shutdown_connections 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 --2- 172.21.15.1:0/2615782318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e400b4e40 0x7f1e40161f00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 --2- 172.21.15.1:0/2615782318 >> v2:172.21.15.3:3300/0 conn(0x7f1e401615f0 0x7f1e400d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 -- 172.21.15.1:0/2615782318 >> 172.21.15.1:0/2615782318 conn(0x7f1e400d3ef0 msgr2=0x7f1e400d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:49.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 -- 172.21.15.1:0/2615782318 shutdown_connections 2024-03-24T10:35:49.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.739+0000 7f1e4754f640 1 -- 172.21.15.1:0/2615782318 wait complete. 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.740+0000 7f1e4754f640 1 Processor -- start 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.740+0000 7f1e4754f640 1 -- start start 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 0x7f1e400b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1e40155ed0 0x7f1e4015ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1e400c8e50 con 0x7f1e401619d0 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f1e400b8570 con 0x7f1e401615f0 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e4754f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1e40156380 con 0x7f1e40155ed0 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e44aea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e44aea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53314/0 (socket says 172.21.15.1:53314) 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e44aea640 1 -- 172.21.15.1:0/4118111063 learned_addr learned my addr 172.21.15.1:0/4118111063 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e452eb640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 0x7f1e400b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.741+0000 7f1e45aec640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1e40155ed0 0x7f1e4015ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 -- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1e40155ed0 msgr2=0x7f1e4015ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1e40155ed0 0x7f1e4015ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 -- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 msgr2=0x7f1e400b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 0x7f1e400b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 -- 172.21.15.1:0/4118111063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e340094d0 con 0x7f1e401619d0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e452eb640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 0x7f1e400b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e44aea640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1e34000c00 tx=0x7f1e340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3403abe0 con 0x7f1e401619d0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3403ad40 con 0x7f1e401619d0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e400c82d0 con 0x7f1e401619d0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34044610 con 0x7f1e401619d0 2024-03-24T10:35:49.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.742+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e400c8890 con 0x7f1e401619d0 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.743+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4004f470 con 0x7f1e401619d0 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.744+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e34044770 con 0x7f1e401619d0 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.744+0000 7f1e3dffb640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 0x7f1e2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.744+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5227+0+0 (secure 0 0 0) 0x7f1e3402d040 con 0x7f1e401619d0 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.744+0000 7f1e452eb640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 0x7f1e2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:49.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.744+0000 7f1e452eb640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 0x7f1e2c06dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1e38006fd0 tx=0x7f1e38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:49.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.749+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e34077220 con 0x7f1e401619d0 2024-03-24T10:35:49.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:49.928+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1e4008d640 con 0x7f1e401619d0 2024-03-24T10:35:50.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.612+0000 7f1e3dffb640 1 -- 172.21.15.1:0/4118111063 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v48) ==== 170+0+0 (secure 0 0 0) 0x7f1e3407a8e0 con 0x7f1e401619d0 2024-03-24T10:35:50.613 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-24T10:35:50.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 msgr2=0x7f1e2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 0x7f1e2c06dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1e38006fd0 tx=0x7f1e38008040 comp rx=0 tx=0).stop 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 msgr2=0x7f1e400c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1e34000c00 tx=0x7f1e340046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 shutdown_connections 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f1e2c06b810 0x7f1e2c06dcd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1e40155ed0 0x7f1e4015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e401619d0 0x7f1e400c7bd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 --2- 172.21.15.1:0/4118111063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e401615f0 0x7f1e400b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 >> 172.21.15.1:0/4118111063 conn(0x7f1e400d3ef0 msgr2=0x7f1e400d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 shutdown_connections 2024-03-24T10:35:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.616+0000 7f1e4754f640 1 -- 172.21.15.1:0/4118111063 wait complete. 2024-03-24T10:35:50.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T10:35:50.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 Processor -- start 2024-03-24T10:35:50.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 -- start start 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5a00d1e20 0x7ff5a00cf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a00b6f50 con 0x7ff5a00d2540 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a00b7230 con 0x7ff5a00b7d10 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.753+0000 7ff5a6177640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a00cfeb0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5a00d2540 0x7ff5a00d2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49856/0 (socket says 172.21.15.1:49856) 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59f7fe640 1 -- 172.21.15.1:0/1619342959 learned_addr learned my addr 172.21.15.1:0/1619342959 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59effd640 1 --2- 172.21.15.1:0/1619342959 >> v2:172.21.15.1:3300/0 conn(0x7ff5a00d1e20 0x7ff5a00cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59e7fc640 1 -- 172.21.15.1:0/1619342959 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104255803 0 0) 0x7ff5a00b6f50 con 0x7ff5a00d2540 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59e7fc640 1 -- 172.21.15.1:0/1619342959 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff588003520 con 0x7ff5a00d2540 2024-03-24T10:35:50.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59effd640 1 -- 172.21.15.1:0/1619342959 >> v1:172.21.15.3:6789/0 conn(0x7ff5a00b7d10 legacy=0x7ff5a00b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59effd640 1 -- 172.21.15.1:0/1619342959 >> v1:172.21.15.1:6789/0 conn(0x7ff5a00d2540 legacy=0x7ff5a00d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.754+0000 7ff59effd640 1 -- 172.21.15.1:0/1619342959 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a00b7510 con 0x7ff5a00d1e20 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff59effd640 1 --2- 172.21.15.1:0/1619342959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a00cf560 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7fccf0c0d36249 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff59e7fc640 1 -- 172.21.15.1:0/1619342959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59402f810 con 0x7ff5a00d1e20 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff59e7fc640 1 -- 172.21.15.1:0/1619342959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59402f970 con 0x7ff5a00d1e20 2024-03-24T10:35:50.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff59e7fc640 1 -- 172.21.15.1:0/1619342959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59402fd00 con 0x7ff5a00d1e20 2024-03-24T10:35:50.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff5a6177640 1 -- 172.21.15.1:0/1619342959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 msgr2=0x7ff5a00cf560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff5a6177640 1 --2- 172.21.15.1:0/1619342959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a00cf560 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff5a6177640 1 -- 172.21.15.1:0/1619342959 shutdown_connections 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff5a6177640 1 --2- 172.21.15.1:0/1619342959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a00cf560 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.755+0000 7ff5a6177640 1 -- 172.21.15.1:0/1619342959 >> 172.21.15.1:0/1619342959 conn(0x7ff5a009d1d0 msgr2=0x7ff5a00d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.756+0000 7ff5a6177640 1 -- 172.21.15.1:0/1619342959 shutdown_connections 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.756+0000 7ff5a6177640 1 -- 172.21.15.1:0/1619342959 wait complete. 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.757+0000 7ff5a6177640 1 Processor -- start 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.757+0000 7ff5a6177640 1 -- start start 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a00b7d10 0x7ff5a01e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 0x7ff5a006c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a00b22a0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01e3910 con 0x7ff5a00b7d10 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a00c9500 con 0x7ff5a00d2540 2024-03-24T10:35:50.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38488/0 (socket says 172.21.15.1:38488) 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 -- 172.21.15.1:0/4273956807 learned_addr learned my addr 172.21.15.1:0/4273956807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59f7fe640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a00b7d10 0x7ff5a01e33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59ffff640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 0x7ff5a006c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 -- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 msgr2=0x7ff5a006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 0x7ff5a006c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 -- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a00b7d10 msgr2=0x7ff5a01e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a00b7d10 0x7ff5a01e33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 -- 172.21.15.1:0/4273956807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff594009400 con 0x7ff5a00d1e20 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59ffff640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 0x7ff5a006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff59effd640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff59402c000 tx=0x7ff594004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59403c040 con 0x7ff5a00d1e20 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5940378f0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a00c8c60 con 0x7ff5a00d1e20 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.758+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594037ca0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.759+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a00c9220 con 0x7ff5a00d1e20 2024-03-24T10:35:50.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.759+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a00b23e0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.760+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff59402d3a0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.760+0000 7ff587fff640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 0x7ff58c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.761+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5609+0+0 (secure 0 0 0) 0x7ff5940abec0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.761+0000 7ff59f7fe640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 0x7ff58c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:50.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.761+0000 7ff59f7fe640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 0x7ff58c06dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff5900011e0 tx=0x7ff590007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:50.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.765+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5a00b23e0 con 0x7ff5a00d1e20 2024-03-24T10:35:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:50.981+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff5a00b23e0 con 0x7ff5a00d1e20 2024-03-24T10:35:51.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.615+0000 7ff587fff640 1 -- 172.21.15.1:0/4273956807 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v23) ==== 172+0+0 (secure 0 0 0) 0x7ff5a00b23e0 con 0x7ff5a00d1e20 2024-03-24T10:35:51.615 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 7 and data pool 8 2024-03-24T10:35:51.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.623+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 msgr2=0x7ff58c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.623+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 0x7ff58c06dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff5900011e0 tx=0x7ff590007d30 comp rx=0 tx=0).stop 2024-03-24T10:35:51.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.623+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 msgr2=0x7ff5a006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.623+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff59402c000 tx=0x7ff594004730 comp rx=0 tx=0).stop 2024-03-24T10:35:51.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 shutdown_connections 2024-03-24T10:35:51.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff58c06b810 0x7ff58c06dcd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff5a00d2540 0x7ff5a006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5a00d1e20 0x7ff5a006bb50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 --2- 172.21.15.1:0/4273956807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a00b7d10 0x7ff5a01e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.624+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 >> 172.21.15.1:0/4273956807 conn(0x7ff5a009d1d0 msgr2=0x7ff5a00cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:51.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.628+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 shutdown_connections 2024-03-24T10:35:51.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.630+0000 7ff5a6177640 1 -- 172.21.15.1:0/4273956807 wait complete. 2024-03-24T10:35:51.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.764+0000 7f37875f6640 1 Processor -- start 2024-03-24T10:35:51.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.764+0000 7f37875f6640 1 -- start start 2024-03-24T10:35:51.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.764+0000 7f37875f6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f37875f6640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f37875f6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37800d2e10 con 0x7f37800b5020 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f37875f6640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f37800cf1a0 con 0x7f378015e530 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f37875f6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f37800cf480 con 0x7f378015e150 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3784b91640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3784b91640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:59106/0 (socket says 172.21.15.1:59106) 2024-03-24T10:35:51.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:40290/0 (socket says 172.21.15.1:40290) 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785b93640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f37800b5020 0x7f3780161e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:53680/0 (socket says 172.21.15.1:53680) 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3784b91640 1 -- 172.21.15.1:0/3825630785 learned_addr learned my addr 172.21.15.1:0/3825630785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 -- 172.21.15.1:0/3825630785 >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 msgr2=0x7f37800b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 --2- 172.21.15.1:0/3825630785 >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 -- 172.21.15.1:0/3825630785 >> v1:172.21.15.3:6789/0 conn(0x7f37800b5020 legacy=0x7f3780161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3785392640 1 -- 172.21.15.1:0/3825630785 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37800cf760 con 0x7f378015e150 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.765+0000 7f3784b91640 1 --2- 172.21.15.1:0/3825630785 >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.766+0000 7f3785392640 1 --2- 172.21.15.1:0/3825630785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f377c009d90 tx=0x7f377c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abe3a221abc6bfb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.766+0000 7f3777fff640 1 -- 172.21.15.1:0/3825630785 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c02cc50 con 0x7f378015e150 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.766+0000 7f3777fff640 1 -- 172.21.15.1:0/3825630785 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c02cdb0 con 0x7f378015e150 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.766+0000 7f3777fff640 1 -- 172.21.15.1:0/3825630785 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c034660 con 0x7f378015e150 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 -- 172.21.15.1:0/3825630785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 msgr2=0x7f37800ceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 --2- 172.21.15.1:0/3825630785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f377c009d90 tx=0x7f377c02bea0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 -- 172.21.15.1:0/3825630785 shutdown_connections 2024-03-24T10:35:51.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 --2- 172.21.15.1:0/3825630785 >> v2:172.21.15.3:3300/0 conn(0x7f378015e530 0x7f37800b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 --2- 172.21.15.1:0/3825630785 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ceac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 -- 172.21.15.1:0/3825630785 >> 172.21.15.1:0/3825630785 conn(0x7f378009d1b0 msgr2=0x7f37800d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 -- 172.21.15.1:0/3825630785 shutdown_connections 2024-03-24T10:35:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.767+0000 7f37875f6640 1 -- 172.21.15.1:0/3825630785 wait complete. 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.768+0000 7f37875f6640 1 Processor -- start 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.768+0000 7f37875f6640 1 -- start start 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378015e530 0x7f37800ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f37801f8320 con 0x7f37800b5020 2024-03-24T10:35:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f37801f77a0 con 0x7f378015e530 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f37875f6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f37801f7a80 con 0x7f378015e150 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38494/0 (socket says 172.21.15.1:38494) 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 -- 172.21.15.1:0/3505438323 learned_addr learned my addr 172.21.15.1:0/3505438323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3784b91640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785b93640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378015e530 0x7f37800ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 -- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 msgr2=0x7f37800ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 -- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378015e530 msgr2=0x7f37800ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378015e530 0x7f37800ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.769+0000 7f3785392640 1 -- 172.21.15.1:0/3505438323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37700094d0 con 0x7f37800b5020 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f3785392640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f377c009d90 tx=0x7f377c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c040650 con 0x7f37800b5020 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c0407b0 con 0x7f37800b5020 2024-03-24T10:35:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c040b60 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377c009a70 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.770+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801f8040 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.771+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378004f4e0 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.771+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f377c040cc0 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.772+0000 7f3775ffb640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 0x7f376406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.772+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5709+0+0 (secure 0 0 0) 0x7f377c0acef0 con 0x7f37800b5020 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.772+0000 7f3784b91640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 0x7f376406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.772+0000 7f3784b91640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 0x7f376406dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f37800c9e70 tx=0x7f3770009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.776+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f377c077ad0 con 0x7f37800b5020 2024-03-24T10:35:51.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.957+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37801f5040 con 0x7f37800b5020 2024-03-24T10:35:51.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.958+0000 7f3775ffb640 1 -- 172.21.15.1:0/3505438323 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+16550 (secure 0 0 0) 0x7f377c07b190 con 0x7f37800b5020 2024-03-24T10:35:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:51.959 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:50.982732+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:51.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 msgr2=0x7f376406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 0x7f376406dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f37800c9e70 tx=0x7f3770009240 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 msgr2=0x7f37801555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f377c009d90 tx=0x7f377c004a40 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 shutdown_connections 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f376406b810 0x7f376406dcd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378015e530 0x7f37800ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f378015e150 0x7f37800ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 --2- 172.21.15.1:0/3505438323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37800b5020 0x7f37801555f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 >> 172.21.15.1:0/3505438323 conn(0x7f378009d1b0 msgr2=0x7f378015d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 shutdown_connections 2024-03-24T10:35:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:51.960+0000 7f37875f6640 1 -- 172.21.15.1:0/3505438323 wait complete. 2024-03-24T10:35:51.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:52.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.073+0000 7f42fe6da640 1 Processor -- start 2024-03-24T10:35:52.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.073+0000 7f42fe6da640 1 -- start start 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42fe6da640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42f806f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42fe6da640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f80b2f30 con 0x7f42f806f0d0 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42fe6da640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f80b3180 con 0x7f42f8146ee0 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42fe6da640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42f806fa50 con 0x7f42f806eaa0 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42f77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42f806f0d0 0x7f42f806e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49866/0 (socket says 172.21.15.1:49866) 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42f77fe640 1 -- 172.21.15.1:0/3643109981 learned_addr learned my addr 172.21.15.1:0/3643109981 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:52.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42f806f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400314634 0 0) 0x7f42f80b2f30 con 0x7f42f806f0d0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.074+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e4003520 con 0x7f42f806f0d0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 842283786 0 0) 0x7f42e4003520 con 0x7f42f806f0d0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 msgr2=0x7f42f806f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42f806f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 >> v1:172.21.15.3:6789/0 conn(0x7f42f8146ee0 legacy=0x7f42f8071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f80b3460 con 0x7f42f806f0d0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f42e0001670 con 0x7f42f806f0d0 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 >> v1:172.21.15.1:6789/0 conn(0x7f42f806f0d0 legacy=0x7f42f806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42e40039a0 0x7f42e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42e40047c0 0x7f42e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42e4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42e400bb40 con 0x7f42e40039a0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f42e400bcb0 con 0x7f42e40047c0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42e400be70 con 0x7f42f806eaa0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42e40039a0 0x7f42e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f77fe640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42e40047c0 0x7f42e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42e4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 -- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 msgr2=0x7f42e4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42e4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 -- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42e40047c0 msgr2=0x7f42e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42e40047c0 0x7f42e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f7fff640 1 -- 172.21.15.1:0/3643109981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e4003520 con 0x7f42e40039a0 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42f806eaa0 0x7f42e4005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.075+0000 7f42f77fe640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42e40047c0 0x7f42e4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42e40039a0 0x7f42e4003dc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f42e8001750 tx=0x7f42e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894212337c6bf5b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80052d0 con 0x7f42e40039a0 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e802e050 con 0x7f42e40039a0 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42f6ffd640 1 -- 172.21.15.1:0/3643109981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e80035f0 con 0x7f42e40039a0 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42fe6da640 1 -- 172.21.15.1:0/3643109981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42e40039a0 msgr2=0x7f42e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3643109981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42e40039a0 0x7f42e4003dc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f42e8001750 tx=0x7f42e40057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42fcc77640 1 -- 172.21.15.1:0/3643109981 reap_dead start 2024-03-24T10:35:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42fe6da640 1 -- 172.21.15.1:0/3643109981 shutdown_connections 2024-03-24T10:35:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.076+0000 7f42fe6da640 1 -- 172.21.15.1:0/3643109981 >> 172.21.15.1:0/3643109981 conn(0x7f42f80cff00 msgr2=0x7f42f80d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.077+0000 7f42fe6da640 1 -- 172.21.15.1:0/3643109981 shutdown_connections 2024-03-24T10:35:52.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.077+0000 7f42fe6da640 1 -- 172.21.15.1:0/3643109981 wait complete. 2024-03-24T10:35:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 Processor -- start 2024-03-24T10:35:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 -- start start 2024-03-24T10:35:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 0x7f42f8144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 0x7f42f81455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.078+0000 7f42fe6da640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42f81f8620 con 0x7f42f8144d30 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fe6da640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f42f80cca40 con 0x7f42f81f8190 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fe6da640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42f8144580 con 0x7f42e40039a0 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38518/0 (socket says 172.21.15.1:38518) 2024-03-24T10:35:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 -- 172.21.15.1:0/3690730709 learned_addr learned my addr 172.21.15.1:0/3690730709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f77fe640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 0x7f42f81455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 0x7f42f8144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 -- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 msgr2=0x7f42f8144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 0x7f42f8144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 -- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 msgr2=0x7f42f81455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 0x7f42f81455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 -- 172.21.15.1:0/3690730709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f80cf6f0 con 0x7f42f8144d30 2024-03-24T10:35:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 0x7f42f8144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f77fe640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 0x7f42f81455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fcc77640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009ed0 tx=0x7f42f0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f000e050 con 0x7f42f8144d30 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f000f040 con 0x7f42f8144d30 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f80cf130 con 0x7f42f8144d30 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0003bd0 con 0x7f42f8144d30 2024-03-24T10:35:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.079+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f8144750 con 0x7f42f8144d30 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.080+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f80ccbd0 con 0x7f42f8144d30 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.081+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42f0015440 con 0x7f42f8144d30 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.081+0000 7f42f4ff9640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 0x7f42d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.081+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5709+0+0 (secure 0 0 0) 0x7f42f008a6a0 con 0x7f42f8144d30 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.081+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 0x7f42d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.082+0000 7f42f7fff640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 0x7f42d806dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42e8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.085+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42f0055150 con 0x7f42f8144d30 2024-03-24T10:35:52.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.268+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42f80672a0 con 0x7f42f8144d30 2024-03-24T10:35:52.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.269+0000 7f42f4ff9640 1 -- 172.21.15.1:0/3690730709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+16550 (secure 0 0 0) 0x7f42f0058810 con 0x7f42f8144d30 2024-03-24T10:35:52.270 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:52.271 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:50.982732+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 msgr2=0x7f42d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 0x7f42d806dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42e8009030 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 msgr2=0x7f42f8145b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009ed0 tx=0x7f42f0010590 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 shutdown_connections 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42d806b810 0x7f42d806dcd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42f8144d30 0x7f42f8145b20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42f81f8190 0x7f42f81455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 --2- 172.21.15.1:0/3690730709 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42e40039a0 0x7f42f8144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 >> 172.21.15.1:0/3690730709 conn(0x7f42f80cff00 msgr2=0x7f42f80b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 shutdown_connections 2024-03-24T10:35:52.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.272+0000 7f42fe6da640 1 -- 172.21.15.1:0/3690730709 wait complete. 2024-03-24T10:35:52.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T10:35:52.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.388+0000 7f2592fc7640 1 Processor -- start 2024-03-24T10:35:52.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.388+0000 7f2592fc7640 1 -- start start 2024-03-24T10:35:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2592fc7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f258c0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2592fc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258c0cdac0 con 0x7f258c0d08f0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2592fc7640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258c0cd7e0 con 0x7f258c0cd220 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2592fc7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f258c0cfd90 con 0x7f258c15bfa0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2591564640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f258c0d08f0 0x7f258c0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49882/0 (socket says 172.21.15.1:49882) 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2591564640 1 -- 172.21.15.1:0/1008703017 learned_addr learned my addr 172.21.15.1:0/1008703017 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2590d63640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f258c0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254045796 0 0) 0x7f258c0cdac0 con 0x7f258c0d08f0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f257c003520 con 0x7f258c0d08f0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4191291276 0 0) 0x7f258c0cd7e0 con 0x7f258c0cd220 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f258c0cdac0 con 0x7f258c0cd220 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 26933002 0 0) 0x7f257c003520 con 0x7f258c0d08f0 2024-03-24T10:35:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 msgr2=0x7f258c0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f258c0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 >> v1:172.21.15.3:6789/0 conn(0x7f258c0cd220 legacy=0x7f258c0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c0b1210 con 0x7f258c0d08f0 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f25840019f0 con 0x7f258c0d08f0 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f2590d63640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f258c0d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 >> v1:172.21.15.1:6789/0 conn(0x7f258c0d08f0 legacy=0x7f258c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.389+0000 7f258b7fe640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f257c003b30 0x7f257c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258b7fe640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f257c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258b7fe640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f257c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f258c0cfd90 con 0x7f257c0047c0 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f257c00ba30 con 0x7f257c003b30 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f257c00bbc0 con 0x7f258c15bfa0 2024-03-24T10:35:52.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f257c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f2590d63640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f257c003b30 0x7f257c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f2591564640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f257c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 -- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 msgr2=0x7f257c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c15bfa0 0x7f257c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 -- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f257c003b30 msgr2=0x7f257c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f257c003b30 0x7f257c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.390+0000 7f258bfff640 1 -- 172.21.15.1:0/1008703017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2580009400 con 0x7f257c0047c0 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f258bfff640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f257c004c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f25780018a0 tx=0x7f257c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faaf13c90dbf6660 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25780052d0 con 0x7f257c0047c0 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2578005430 con 0x7f257c0047c0 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f258b7fe640 1 -- 172.21.15.1:0/1008703017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578003810 con 0x7f257c0047c0 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 -- 172.21.15.1:0/1008703017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 msgr2=0x7f257c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 --2- 172.21.15.1:0/1008703017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f257c004c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f25780018a0 tx=0x7f257c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2591564640 1 -- 172.21.15.1:0/1008703017 reap_dead start 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 -- 172.21.15.1:0/1008703017 shutdown_connections 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 -- 172.21.15.1:0/1008703017 >> 172.21.15.1:0/1008703017 conn(0x7f258c0d1c00 msgr2=0x7f258c0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 -- 172.21.15.1:0/1008703017 shutdown_connections 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.391+0000 7f2592fc7640 1 -- 172.21.15.1:0/1008703017 wait complete. 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.392+0000 7f2592fc7640 1 Processor -- start 2024-03-24T10:35:52.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.392+0000 7f2592fc7640 1 -- start start 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 0x7f258c1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 0x7f258c1f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f258c0b6e60 con 0x7f257c0047c0 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f258c0b7140 con 0x7f258c0b6940 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2592fc7640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f258c0b7420 con 0x7f258c0b64c0 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2590d63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2590d63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38540/0 (socket says 172.21.15.1:38540) 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.393+0000 7f2590d63640 1 -- 172.21.15.1:0/3491282591 learned_addr learned my addr 172.21.15.1:0/3491282591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f258bfff640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 0x7f258c1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2591564640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 0x7f258c1f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 -- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 msgr2=0x7f258c1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 0x7f258c1f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 -- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 msgr2=0x7f258c1f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 0x7f258c1f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 -- 172.21.15.1:0/3491282591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c0b7700 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f258bfff640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 0x7f258c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2591564640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 0x7f258c1f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2590d63640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2580011a90 tx=0x7f2580011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258000fc10 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258000fd70 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258c06e090 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258000fee0 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.394+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258c1f4f60 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.395+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f258c0b78f0 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.396+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2580004260 con 0x7f257c0047c0 2024-03-24T10:35:52.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.396+0000 7f25897fa640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 0x7f257006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.396+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5709+0+0 (secure 0 0 0) 0x7f258008a7f0 con 0x7f257c0047c0 2024-03-24T10:35:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.396+0000 7f258bfff640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 0x7f257006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.396+0000 7f258bfff640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 0x7f257006dcd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2578034000 tx=0x7f2578031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.400+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2580055210 con 0x7f257c0047c0 2024-03-24T10:35:52.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.620+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f258c151b00 con 0x7f257c0047c0 2024-03-24T10:35:52.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.628+0000 7f25897fa640 1 -- 172.21.15.1:0/3491282591 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f25800220b0 con 0x7f257c0047c0 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.630+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 msgr2=0x7f257006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.630+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 0x7f257006dcd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2578034000 tx=0x7f2578031000 comp rx=0 tx=0).stop 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 msgr2=0x7f258c06db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2580011a90 tx=0x7f2580011f50 comp rx=0 tx=0).stop 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 shutdown_connections 2024-03-24T10:35:52.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f257006b810 0x7f257006dcd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f258c0b6940 0x7f258c1f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f258c0b64c0 0x7f258c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 --2- 172.21.15.1:0/3491282591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f257c0047c0 0x7f258c06db50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 >> 172.21.15.1:0/3491282591 conn(0x7f258c0d1c00 msgr2=0x7f258c06eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 shutdown_connections 2024-03-24T10:35:52.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.631+0000 7f2592fc7640 1 -- 172.21.15.1:0/3491282591 wait complete. 2024-03-24T10:35:52.643 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T10:35:52.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T10:35:52.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 Processor -- start 2024-03-24T10:35:52.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 -- start start 2024-03-24T10:35:52.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66a00b4e40 0x7f66a00d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a00c7a40 con 0x7f66a01615f0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a00b55b0 con 0x7f66a0162610 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.748+0000 7f66a4a55640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f66a00cfd50 con 0x7f66a00b4e40 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f66a01615f0 0x7f66a0161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49896/0 (socket says 172.21.15.1:49896) 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669e59c640 1 -- 172.21.15.1:0/2626837115 learned_addr learned my addr 172.21.15.1:0/2626837115 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f6697fff640 1 --2- 172.21.15.1:0/2626837115 >> v2:172.21.15.3:3300/0 conn(0x7f66a00b4e40 0x7f66a00d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220803433 0 0) 0x7f66a00c7a40 con 0x7f66a01615f0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f668c003520 con 0x7f66a01615f0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821211210 0 0) 0x7f66a00b55b0 con 0x7f66a0162610 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a00c7a40 con 0x7f66a0162610 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2849586147 0 0) 0x7f668c003520 con 0x7f66a01615f0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 >> v2:172.21.15.3:3300/0 conn(0x7f66a00b4e40 msgr2=0x7f66a00d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 --2- 172.21.15.1:0/2626837115 >> v2:172.21.15.3:3300/0 conn(0x7f66a00b4e40 0x7f66a00d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 >> v1:172.21.15.3:6789/0 conn(0x7f66a0162610 legacy=0x7f66a00d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a00b52d0 con 0x7f66a01615f0 2024-03-24T10:35:52.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f6690001670 con 0x7f66a01615f0 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 >> v1:172.21.15.1:6789/0 conn(0x7f66a01615f0 legacy=0x7f66a0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 0x7f668c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f668c0047c0 0x7f668c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c005180 0x7f668c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66a00cfd50 con 0x7f668c003b50 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f668c00c050 con 0x7f668c0047c0 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f668c00c210 con 0x7f668c005180 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669e59c640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 0x7f668c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f6697fff640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f668c0047c0 0x7f668c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.749+0000 7f669ed9d640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c005180 0x7f668c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 -- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c005180 msgr2=0x7f668c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c005180 0x7f668c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 -- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f668c0047c0 msgr2=0x7f668c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f668c0047c0 0x7f668c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 -- 172.21.15.1:0/2626837115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c003520 con 0x7f668c003b50 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669ed9d640 1 -- 172.21.15.1:0/2626837115 reap_dead start 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669e59c640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 0x7f668c003f50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6690007f90 tx=0x7f668c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fca0933531dceb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690005ad0 con 0x7f668c003b50 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6690005c30 con 0x7f668c003b50 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.750+0000 7f669dd9b640 1 -- 172.21.15.1:0/2626837115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669002c9b0 con 0x7f668c003b50 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 -- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 msgr2=0x7f668c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 0x7f668c003f50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6690007f90 tx=0x7f668c0057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 -- 172.21.15.1:0/2626837115 shutdown_connections 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 --2- 172.21.15.1:0/2626837115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f668c003b50 0x7f668c003f50 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 -- 172.21.15.1:0/2626837115 >> 172.21.15.1:0/2626837115 conn(0x7f66a00d3ef0 msgr2=0x7f66a00d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 -- 172.21.15.1:0/2626837115 shutdown_connections 2024-03-24T10:35:52.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.751+0000 7f66a4a55640 1 -- 172.21.15.1:0/2626837115 wait complete. 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.752+0000 7f66a4a55640 1 Processor -- start 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.752+0000 7f66a4a55640 1 -- start start 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c003b50 0x7f66a00b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66a00b81e0 0x7f66a014b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66a00cdfd0 con 0x7f66a00cdb40 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f66a00ccc80 con 0x7f66a00b81e0 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f66a4a55640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f66a00cc9a0 con 0x7f668c003b50 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38554/0 (socket says 172.21.15.1:38554) 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 -- 172.21.15.1:0/642885135 learned_addr learned my addr 172.21.15.1:0/642885135 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 -- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c003b50 msgr2=0x7f66a00b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c003b50 0x7f66a00b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 -- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66a00b81e0 msgr2=0x7f66a014b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f6697fff640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66a00b81e0 0x7f66a014b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66a00b81e0 0x7f66a014b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.753+0000 7f669ed9d640 1 -- 172.21.15.1:0/642885135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a00cd060 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f669ed9d640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f66a00b9fa0 tx=0x7f669800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669800ee50 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6698003b80 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a00cd340 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980050d0 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.754+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a00cc630 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.755+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a00cd860 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.755+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6698005230 con 0x7f66a00cdb40 2024-03-24T10:35:52.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.756+0000 7f6696ffd640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:52.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.756+0000 7f669e59c640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:52.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.756+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f669808a020 con 0x7f66a00cdb40 2024-03-24T10:35:52.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.756+0000 7f669e59c640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 0x7f668006dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6690034000 tx=0x7f6690039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:52.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.762+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66a00cd860 con 0x7f66a00cdb40 2024-03-24T10:35:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:52.980+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f66a00cd860 con 0x7f66a00cdb40 2024-03-24T10:35:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.629+0000 7f6696ffd640 1 -- 172.21.15.1:0/642885135 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f66a00cd860 con 0x7f66a00cdb40 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 msgr2=0x7f668006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 0x7f668006dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6690034000 tx=0x7f6690039040 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 msgr2=0x7f66a00cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f66a00b9fa0 tx=0x7f669800ec00 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 shutdown_connections 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66a00cdb40 0x7f66a00cffb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66a00b81e0 0x7f66a014b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 --2- 172.21.15.1:0/642885135 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f668c003b50 0x7f66a00b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 >> 172.21.15.1:0/642885135 conn(0x7f66a00d3ef0 msgr2=0x7f66a00d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.631+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 shutdown_connections 2024-03-24T10:35:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.632+0000 7f66a4a55640 1 -- 172.21.15.1:0/642885135 wait complete. 2024-03-24T10:35:53.643 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T10:35:53.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.744+0000 7f6ada46f640 1 Processor -- start 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.744+0000 7f6ada46f640 1 -- start start 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e0a0 0x7f6ad40d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ad40cebc0 0x7f6ad40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 0x7f6ad40d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad40d0160 con 0x7f6ad40cebc0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad40d4eb0 con 0x7f6ad415e480 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ada46f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad40cf0e0 con 0x7f6ad415e0a0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ad37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ad40cebc0 0x7f6ad40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ad37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ad40cebc0 0x7f6ad40d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38558/0 (socket says 172.21.15.1:38558) 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.745+0000 7f6ad37fe640 1 -- 172.21.15.1:0/561644652 learned_addr learned my addr 172.21.15.1:0/561644652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 -- 172.21.15.1:0/561644652 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e0a0 msgr2=0x7f6ad40d4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/561644652 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e0a0 0x7f6ad40d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 -- 172.21.15.1:0/561644652 >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 msgr2=0x7f6ad40d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad8a0c640 1 --2- 172.21.15.1:0/561644652 >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 0x7f6ad40d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/561644652 >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 0x7f6ad40d2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 -- 172.21.15.1:0/561644652 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad40cf3c0 con 0x7f6ad40cebc0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad8a0c640 1 --2- 172.21.15.1:0/561644652 >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 0x7f6ad40d2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/561644652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40d2590 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40ba030 tx=0x7f6ac402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a73281312cbab46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.746+0000 7f6ad2ffd640 1 -- 172.21.15.1:0/561644652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4030860 con 0x7f6ad40cebc0 2024-03-24T10:35:53.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ad2ffd640 1 -- 172.21.15.1:0/561644652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac40309c0 con 0x7f6ad40cebc0 2024-03-24T10:35:53.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ad2ffd640 1 -- 172.21.15.1:0/561644652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4030d50 con 0x7f6ad40cebc0 2024-03-24T10:35:53.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 -- 172.21.15.1:0/561644652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 msgr2=0x7f6ad40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 --2- 172.21.15.1:0/561644652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40d2590 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40ba030 tx=0x7f6ac402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 -- 172.21.15.1:0/561644652 shutdown_connections 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 --2- 172.21.15.1:0/561644652 >> v2:172.21.15.3:3300/0 conn(0x7f6ad415e480 0x7f6ad40d2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 --2- 172.21.15.1:0/561644652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40d2590 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 --2- 172.21.15.1:0/561644652 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e0a0 0x7f6ad40d4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 -- 172.21.15.1:0/561644652 >> 172.21.15.1:0/561644652 conn(0x7f6ad409d1c0 msgr2=0x7f6ad40d3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 -- 172.21.15.1:0/561644652 shutdown_connections 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.747+0000 7f6ada46f640 1 -- 172.21.15.1:0/561644652 wait complete. 2024-03-24T10:35:53.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.748+0000 7f6ada46f640 1 Processor -- start 2024-03-24T10:35:53.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.748+0000 7f6ada46f640 1 -- start start 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ad415e0a0 0x7f6ad41f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e480 0x7f6ad41f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41f8000 con 0x7f6ad40cebc0 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad40b50d0 con 0x7f6ad415e0a0 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ada46f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad41f8230 con 0x7f6ad415e480 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ad3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ad3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38560/0 (socket says 172.21.15.1:38560) 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ad3fff640 1 -- 172.21.15.1:0/978287693 learned_addr learned my addr 172.21.15.1:0/978287693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.749+0000 7f6ad8a0c640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e480 0x7f6ad41f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 -- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e480 msgr2=0x7f6ad41f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e480 0x7f6ad41f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ad415e0a0 0x7f6ad41f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 -- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ad415e0a0 msgr2=0x7f6ad41f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ad415e0a0 0x7f6ad41f7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 -- 172.21.15.1:0/978287693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac40094d0 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad3fff640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b4a0 tx=0x7f6ac800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8010610 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac8010770 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad41f8510 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8010b20 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.750+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad40b5350 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.751+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad40b5540 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.751+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6ac8010c80 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.752+0000 7f6ad0ff9640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 0x7f6ac006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.752+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f6ac8089e80 con 0x7f6ad40cebc0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.752+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 0x7f6ac006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:53.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.752+0000 7f6ad37fe640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 0x7f6ac006dcd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4036000 tx=0x7f6ac403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:53.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.756+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6ac80548a0 con 0x7f6ad40cebc0 2024-03-24T10:35:53.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.975+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ad404f4e0 con 0x7f6ad40cebc0 2024-03-24T10:35:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.975+0000 7f6ad0ff9640 1 -- 172.21.15.1:0/978287693 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f6ac801e090 con 0x7f6ad40cebc0 2024-03-24T10:35:53.976 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:53.976 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:35:53.976 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 msgr2=0x7f6ac006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 0x7f6ac006dcd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4036000 tx=0x7f6ac403b040 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 msgr2=0x7f6ad40b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b4a0 tx=0x7f6ac800b960 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 shutdown_connections 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6ac006b810 0x7f6ac006dcd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6ad415e480 0x7f6ad41f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ad415e0a0 0x7f6ad41f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 --2- 172.21.15.1:0/978287693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ad40cebc0 0x7f6ad40b37e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 >> 172.21.15.1:0/978287693 conn(0x7f6ad409d1c0 msgr2=0x7f6ad40d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 shutdown_connections 2024-03-24T10:35:53.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:53.977+0000 7f6ada46f640 1 -- 172.21.15.1:0/978287693 wait complete. 2024-03-24T10:35:53.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:35:54.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.090+0000 7fd5e9481640 1 Processor -- start 2024-03-24T10:35:54.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.090+0000 7fd5e9481640 1 -- start start 2024-03-24T10:35:54.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e9481640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e9481640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5e40d1000 0x7fd5e40d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e9481640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e415e770 con 0x7fd5e40b3320 2024-03-24T10:35:54.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e9481640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e40b3190 con 0x7fd5e40d1000 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e9481640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e40d4e40 con 0x7fd5e415e390 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e27fc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5e40d1000 0x7fd5e40d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:40398/0 (socket says 172.21.15.1:40398) 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 -- 172.21.15.1:0/3067710832 learned_addr learned my addr 172.21.15.1:0/3067710832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e37fe640 1 --1- 172.21.15.1:0/3067710832 >> v1:172.21.15.3:6789/0 conn(0x7fd5e40b3320 0x7fd5e40cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:53716/0 (socket says 172.21.15.1:53716) 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 -- 172.21.15.1:0/3067710832 >> v2:172.21.15.3:3300/0 conn(0x7fd5e40d1000 msgr2=0x7fd5e40d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 --2- 172.21.15.1:0/3067710832 >> v2:172.21.15.3:3300/0 conn(0x7fd5e40d1000 0x7fd5e40d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 -- 172.21.15.1:0/3067710832 >> v1:172.21.15.3:6789/0 conn(0x7fd5e40b3320 legacy=0x7fd5e40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.091+0000 7fd5e2ffd640 1 -- 172.21.15.1:0/3067710832 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e40d1e40 con 0x7fd5e415e390 2024-03-24T10:35:54.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.092+0000 7fd5e2ffd640 1 --2- 172.21.15.1:0/3067710832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009870 tx=0x7fd5d402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8592e5ee4821955a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.092+0000 7fd5e1ffb640 1 -- 172.21.15.1:0/3067710832 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40317c0 con 0x7fd5e415e390 2024-03-24T10:35:54.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.092+0000 7fd5e1ffb640 1 -- 172.21.15.1:0/3067710832 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d4031920 con 0x7fd5e415e390 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.092+0000 7fd5e1ffb640 1 -- 172.21.15.1:0/3067710832 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4031cb0 con 0x7fd5e415e390 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 -- 172.21.15.1:0/3067710832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 msgr2=0x7fd5e40d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 --2- 172.21.15.1:0/3067710832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009870 tx=0x7fd5d402b860 comp rx=0 tx=0).stop 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 -- 172.21.15.1:0/3067710832 shutdown_connections 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 --2- 172.21.15.1:0/3067710832 >> v2:172.21.15.3:3300/0 conn(0x7fd5e40d1000 0x7fd5e40d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 --2- 172.21.15.1:0/3067710832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e415e390 0x7fd5e40d24a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 -- 172.21.15.1:0/3067710832 >> 172.21.15.1:0/3067710832 conn(0x7fd5e40d3ec0 msgr2=0x7fd5e40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 -- 172.21.15.1:0/3067710832 shutdown_connections 2024-03-24T10:35:54.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.093+0000 7fd5e9481640 1 -- 172.21.15.1:0/3067710832 wait complete. 2024-03-24T10:35:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.095+0000 7fd5e9481640 1 Processor -- start 2024-03-24T10:35:54.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.095+0000 7fd5e9481640 1 -- start start 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e40b3320 0x7fd5e414f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 0x7fd5e40ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e40b7b60 con 0x7fd5e40d1000 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e415f3d0 con 0x7fd5e415e390 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e9481640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e415f6b0 con 0x7fd5e40b3320 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38564/0 (socket says 172.21.15.1:38564) 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.096+0000 7fd5e27fc640 1 -- 172.21.15.1:0/2846094856 learned_addr learned my addr 172.21.15.1:0/2846094856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e2ffd640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e40b3320 0x7fd5e414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e37fe640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 0x7fd5e40ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 -- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e40b3320 msgr2=0x7fd5e414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e40b3320 0x7fd5e414f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 -- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 msgr2=0x7fd5e40ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 0x7fd5e40ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 -- 172.21.15.1:0/2846094856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d40094d0 con 0x7fd5e40d1000 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e37fe640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 0x7fd5e40ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e27fc640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800e980 tx=0x7fd5d800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8009670 con 0x7fd5e40d1000 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d800ca00 con 0x7fd5e40d1000 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d800cd90 con 0x7fd5e40d1000 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e415f990 con 0x7fd5e40d1000 2024-03-24T10:35:54.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.097+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e414f7a0 con 0x7fd5e40d1000 2024-03-24T10:35:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.099+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e415feb0 con 0x7fd5e40d1000 2024-03-24T10:35:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.099+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd5d8013470 con 0x7fd5e40d1000 2024-03-24T10:35:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.099+0000 7fd5cb7fe640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 0x7fd5d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.099+0000 7fd5e2ffd640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 0x7fd5d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.099+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fd5d808a2b0 con 0x7fd5e40d1000 2024-03-24T10:35:54.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.100+0000 7fd5e2ffd640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 0x7fd5d006dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4036000 tx=0x7fd5d403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.107+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd5d8054e10 con 0x7fd5e40d1000 2024-03-24T10:35:54.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.288+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5e404f4e0 con 0x7fd5e40d1000 2024-03-24T10:35:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.289+0000 7fd5cb7fe640 1 -- 172.21.15.1:0/2846094856 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7fd5d80584d0 con 0x7fd5e40d1000 2024-03-24T10:35:54.290 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:54.290 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.291+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 msgr2=0x7fd5d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.291+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 0x7fd5d006dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4036000 tx=0x7fd5d403d040 comp rx=0 tx=0).stop 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.291+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 msgr2=0x7fd5e415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.291+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800e980 tx=0x7fd5d800ee40 comp rx=0 tx=0).stop 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 shutdown_connections 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd5d006b810 0x7fd5d006dcd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5e415e390 0x7fd5e40ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5e40d1000 0x7fd5e415e910 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 --2- 172.21.15.1:0/2846094856 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5e40b3320 0x7fd5e414f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 >> 172.21.15.1:0/2846094856 conn(0x7fd5e40d3ec0 msgr2=0x7fd5e40d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 shutdown_connections 2024-03-24T10:35:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.292+0000 7fd5e9481640 1 -- 172.21.15.1:0/2846094856 wait complete. 2024-03-24T10:35:54.303 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T10:35:54.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:35:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.408+0000 7f5548e9a640 1 Processor -- start 2024-03-24T10:35:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.408+0000 7f5548e9a640 1 -- start start 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f5548e9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55440d2bb0 0x7f55440b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f5548e9a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55440d1000 0x7f55440b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f5548e9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554415e770 con 0x7f554415e390 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f5548e9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f55440b3190 con 0x7f55440d2bb0 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f5548e9a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f55440d4e40 con 0x7f55440d1000 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f553bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55440d2bb0 0x7f55440b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f554259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f554415e390 0x7f55440d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49906/0 (socket says 172.21.15.1:49906) 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.409+0000 7f554259c640 1 -- 172.21.15.1:0/1287971348 learned_addr learned my addr 172.21.15.1:0/1287971348 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f5542d9d640 1 --2- 172.21.15.1:0/1287971348 >> v2:172.21.15.3:3300/0 conn(0x7f55440d1000 0x7f55440b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f553bfff640 1 -- 172.21.15.1:0/1287971348 >> v2:172.21.15.3:3300/0 conn(0x7f55440d1000 msgr2=0x7f55440b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f553bfff640 1 --2- 172.21.15.1:0/1287971348 >> v2:172.21.15.3:3300/0 conn(0x7f55440d1000 0x7f55440b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f553bfff640 1 -- 172.21.15.1:0/1287971348 >> v1:172.21.15.1:6789/0 conn(0x7f554415e390 legacy=0x7f55440d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f5541d9b640 1 -- 172.21.15.1:0/1287971348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567959765 0 0) 0x7f5534001110 con 0x7f554415e390 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f553bfff640 1 -- 172.21.15.1:0/1287971348 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55440d1e40 con 0x7f55440d2bb0 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.410+0000 7f553bfff640 1 --2- 172.21.15.1:0/1287971348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d2bb0 0x7f55440b3b90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f55440b5960 tx=0x7f552c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97c340d2c8cbfdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5541d9b640 1 -- 172.21.15.1:0/1287971348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c031710 con 0x7f55440d2bb0 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5541d9b640 1 -- 172.21.15.1:0/1287971348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552c02f050 con 0x7f55440d2bb0 2024-03-24T10:35:54.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5541d9b640 1 -- 172.21.15.1:0/1287971348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c031a90 con 0x7f55440d2bb0 2024-03-24T10:35:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 -- 172.21.15.1:0/1287971348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d2bb0 msgr2=0x7f55440b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 --2- 172.21.15.1:0/1287971348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d2bb0 0x7f55440b3b90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f55440b5960 tx=0x7f552c02b7b0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 -- 172.21.15.1:0/1287971348 shutdown_connections 2024-03-24T10:35:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 --2- 172.21.15.1:0/1287971348 >> v2:172.21.15.3:3300/0 conn(0x7f55440d1000 0x7f55440b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 --2- 172.21.15.1:0/1287971348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d2bb0 0x7f55440b3b90 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 -- 172.21.15.1:0/1287971348 >> 172.21.15.1:0/1287971348 conn(0x7f55440d3ec0 msgr2=0x7f55440d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 -- 172.21.15.1:0/1287971348 shutdown_connections 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.411+0000 7f5548e9a640 1 -- 172.21.15.1:0/1287971348 wait complete. 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.412+0000 7f5548e9a640 1 Processor -- start 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.412+0000 7f5548e9a640 1 -- start start 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440d2bb0 0x7f55440b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f554415e390 0x7f55440b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5544156b70 con 0x7f55440d1000 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f55441f3e00 con 0x7f55440d2bb0 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f5548e9a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f55441f4010 con 0x7f554415e390 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f554259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f554259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38580/0 (socket says 172.21.15.1:38580) 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.413+0000 7f554259c640 1 -- 172.21.15.1:0/4020734979 learned_addr learned my addr 172.21.15.1:0/4020734979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:54.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 -- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f554415e390 msgr2=0x7f55440b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f554415e390 0x7f55440b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 -- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440d2bb0 msgr2=0x7f55440b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440d2bb0 0x7f55440b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 -- 172.21.15.1:0/4020734979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c009400 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f554259c640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f553400c430 tx=0x7f553400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55340074e0 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5534007660 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534004a20 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55441f42f0 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.414+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55441f48b0 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.415+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554404f4e0 con 0x7f55440d1000 2024-03-24T10:35:54.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.415+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5534004b80 con 0x7f55440d1000 2024-03-24T10:35:54.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.416+0000 7f553affd640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 0x7f552406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.416+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f5534089ee0 con 0x7f55440d1000 2024-03-24T10:35:54.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.416+0000 7f553bfff640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 0x7f552406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.416+0000 7f553bfff640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 0x7f552406dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f552c035000 tx=0x7f552c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.420+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5534054900 con 0x7f55440d1000 2024-03-24T10:35:54.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.671+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f55440b3320 con 0x7f552406b810 2024-03-24T10:35:54.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.677+0000 7f553affd640 1 -- 172.21.15.1:0/4020734979 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246641 (secure 0 0 0) 0x7f552c031960 con 0x7f552406b810 2024-03-24T10:35:54.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:54.685 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:35:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 msgr2=0x7f552406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 0x7f552406dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f552c035000 tx=0x7f552c030000 comp rx=0 tx=0).stop 2024-03-24T10:35:54.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 msgr2=0x7f5544156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f553400c430 tx=0x7f553400c8f0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 shutdown_connections 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f552406b810 0x7f552406dcd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f554415e390 0x7f55440b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440d2bb0 0x7f55440b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 --2- 172.21.15.1:0/4020734979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55440d1000 0x7f5544156630 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 >> 172.21.15.1:0/4020734979 conn(0x7f55440d3ec0 msgr2=0x7f55440b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 shutdown_connections 2024-03-24T10:35:54.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.680+0000 7f5548e9a640 1 -- 172.21.15.1:0/4020734979 wait complete. 2024-03-24T10:35:54.691 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-03-24T10:35:54.539396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238424,"kb_used_data":5596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736296,"statfs":{"total":773094113280,"available":772849967104,"internally_reserved":0,"allocated":5730304,"data_stored":3045589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931065"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619572+0000","last_change":"2024-03-24T10:35:50.619572+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619572+0000","last_undegraded":"2024-03-24T10:35:50.619572+0000","last_fullsized":"2024-03-24T10:35:50.619572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622464+0000","last_change":"2024-03-24T10:35:50.622464+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622464+0000","last_undegraded":"2024-03-24T10:35:50.622464+0000","last_fullsized":"2024-03-24T10:35:50.622464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627689+0000","last_change":"2024-03-24T10:35:50.627689+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627689+0000","last_undegraded":"2024-03-24T10:35:50.627689+0000","last_fullsized":"2024-03-24T10:35:50.627689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621316+0000","last_change":"2024-03-24T10:35:50.621316+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621316+0000","last_undegraded":"2024-03-24T10:35:50.621316+0000","last_fullsized":"2024-03-24T10:35:50.621316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624565+0000","last_change":"2024-03-24T10:35:50.624565+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624565+0000","last_undegraded":"2024-03-24T10:35:50.624565+0000","last_fullsized":"2024-03-24T10:35:50.624565+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619946+0000","last_change":"2024-03-24T10:35:50.619946+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619946+0000","last_undegraded":"2024-03-24T10:35:50.619946+0000","last_fullsized":"2024-03-24T10:35:50.619946+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624479+0000","last_change":"2024-03-24T10:35:50.624479+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624479+0000","last_undegraded":"2024-03-24T10:35:50.624479+0000","last_fullsized":"2024-03-24T10:35:50.624479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624154+0000","last_change":"2024-03-24T10:35:50.624154+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624154+0000","last_undegraded":"2024-03-24T10:35:50.624154+0000","last_fullsized":"2024-03-24T10:35:50.624154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623889+0000","last_change":"2024-03-24T10:35:50.623889+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623889+0000","last_undegraded":"2024-03-24T10:35:50.623889+0000","last_fullsized":"2024-03-24T10:35:50.623889+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619433+0000","last_change":"2024-03-24T10:35:50.619433+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619433+0000","last_undegraded":"2024-03-24T10:35:50.619433+0000","last_fullsized":"2024-03-24T10:35:50.619433+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619034+0000","last_change":"2024-03-24T10:35:50.619034+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619034+0000","last_undegraded":"2024-03-24T10:35:50.619034+0000","last_fullsized":"2024-03-24T10:35:50.619034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622027+0000","last_change":"2024-03-24T10:35:50.622027+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622027+0000","last_undegraded":"2024-03-24T10:35:50.622027+0000","last_fullsized":"2024-03-24T10:35:50.622027+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619363+0000","last_change":"2024-03-24T10:35:50.619363+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619363+0000","last_undegraded":"2024-03-24T10:35:50.619363+0000","last_fullsized":"2024-03-24T10:35:50.619363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624300+0000","last_change":"2024-03-24T10:35:50.624300+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624300+0000","last_undegraded":"2024-03-24T10:35:50.624300+0000","last_fullsized":"2024-03-24T10:35:50.624300+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621221+0000","last_change":"2024-03-24T10:35:50.621221+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621221+0000","last_undegraded":"2024-03-24T10:35:50.621221+0000","last_fullsized":"2024-03-24T10:35:50.621221+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618425+0000","last_change":"2024-03-24T10:35:50.618425+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618425+0000","last_undegraded":"2024-03-24T10:35:50.618425+0000","last_fullsized":"2024-03-24T10:35:50.618425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627699+0000","last_change":"2024-03-24T10:35:50.627699+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627699+0000","last_undegraded":"2024-03-24T10:35:50.627699+0000","last_fullsized":"2024-03-24T10:35:50.627699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622054+0000","last_change":"2024-03-24T10:35:50.622054+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622054+0000","last_undegraded":"2024-03-24T10:35:50.622054+0000","last_fullsized":"2024-03-24T10:35:50.622054+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627222+0000","last_change":"2024-03-24T10:35:50.627222+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627222+0000","last_undegraded":"2024-03-24T10:35:50.627222+0000","last_fullsized":"2024-03-24T10:35:50.627222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627419+0000","last_change":"2024-03-24T10:35:50.627419+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627419+0000","last_undegraded":"2024-03-24T10:35:50.627419+0000","last_fullsized":"2024-03-24T10:35:50.627419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625486+0000","last_change":"2024-03-24T10:35:50.625486+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625486+0000","last_undegraded":"2024-03-24T10:35:50.625486+0000","last_fullsized":"2024-03-24T10:35:50.625486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622677+0000","last_change":"2024-03-24T10:35:50.622677+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622677+0000","last_undegraded":"2024-03-24T10:35:50.622677+0000","last_fullsized":"2024-03-24T10:35:50.622677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618841+0000","last_change":"2024-03-24T10:35:50.618841+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618841+0000","last_undegraded":"2024-03-24T10:35:50.618841+0000","last_fullsized":"2024-03-24T10:35:50.618841+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622708+0000","last_change":"2024-03-24T10:35:50.622708+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622708+0000","last_undegraded":"2024-03-24T10:35:50.622708+0000","last_fullsized":"2024-03-24T10:35:50.622708+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625709+0000","last_change":"2024-03-24T10:35:50.625709+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625709+0000","last_undegraded":"2024-03-24T10:35:50.625709+0000","last_fullsized":"2024-03-24T10:35:50.625709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625691+0000","last_change":"2024-03-24T10:35:50.625691+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625691+0000","last_undegraded":"2024-03-24T10:35:50.625691+0000","last_fullsized":"2024-03-24T10:35:50.625691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618448+0000","last_change":"2024-03-24T10:35:50.618448+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618448+0000","last_undegraded":"2024-03-24T10:35:50.618448+0000","last_fullsized":"2024-03-24T10:35:50.618448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626365+0000","last_change":"2024-03-24T10:35:50.626365+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626365+0000","last_undegraded":"2024-03-24T10:35:50.626365+0000","last_fullsized":"2024-03-24T10:35:50.626365+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630750+0000","last_change":"2024-03-24T10:35:50.630750+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630750+0000","last_undegraded":"2024-03-24T10:35:50.630750+0000","last_fullsized":"2024-03-24T10:35:50.630750+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621203+0000","last_change":"2024-03-24T10:35:50.621203+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621203+0000","last_undegraded":"2024-03-24T10:35:50.621203+0000","last_fullsized":"2024-03-24T10:35:50.621203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623326+0000","last_change":"2024-03-24T10:35:50.623326+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623326+0000","last_undegraded":"2024-03-24T10:35:50.623326+0000","last_fullsized":"2024-03-24T10:35:50.623326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624597+0000","last_change":"2024-03-24T10:35:50.624597+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624597+0000","last_undegraded":"2024-03-24T10:35:50.624597+0000","last_fullsized":"2024-03-24T10:35:50.624597+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621815+0000","last_change":"2024-03-24T10:35:50.621815+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621815+0000","last_undegraded":"2024-03-24T10:35:50.621815+0000","last_fullsized":"2024-03-24T10:35:50.621815+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638164+0000","last_change":"2024-03-24T10:35:50.635274+0000","last_active":"2024-03-24T10:35:50.638164+0000","last_peered":"2024-03-24T10:35:50.638164+0000","last_clean":"2024-03-24T10:35:50.638164+0000","last_became_active":"2024-03-24T10:35:50.634934+0000","last_became_peered":"2024-03-24T10:35:50.634934+0000","last_unstale":"2024-03-24T10:35:50.638164+0000","last_undegraded":"2024-03-24T10:35:50.638164+0000","last_fullsized":"2024-03-24T10:35:50.638164+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:19.768286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622233+0000","last_change":"2024-03-24T10:35:50.622233+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622233+0000","last_undegraded":"2024-03-24T10:35:50.622233+0000","last_fullsized":"2024-03-24T10:35:50.622233+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634816+0000","last_change":"2024-03-24T10:35:50.633891+0000","last_active":"2024-03-24T10:35:50.634816+0000","last_peered":"2024-03-24T10:35:50.634816+0000","last_clean":"2024-03-24T10:35:50.634816+0000","last_became_active":"2024-03-24T10:35:50.633604+0000","last_became_peered":"2024-03-24T10:35:50.633604+0000","last_unstale":"2024-03-24T10:35:50.634816+0000","last_undegraded":"2024-03-24T10:35:50.634816+0000","last_fullsized":"2024-03-24T10:35:50.634816+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:45.059275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626163+0000","last_change":"2024-03-24T10:35:50.626163+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626163+0000","last_undegraded":"2024-03-24T10:35:50.626163+0000","last_fullsized":"2024-03-24T10:35:50.626163+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638820+0000","last_change":"2024-03-24T10:35:50.637492+0000","last_active":"2024-03-24T10:35:50.638820+0000","last_peered":"2024-03-24T10:35:50.638820+0000","last_clean":"2024-03-24T10:35:50.638820+0000","last_became_active":"2024-03-24T10:35:50.637225+0000","last_became_peered":"2024-03-24T10:35:50.637225+0000","last_unstale":"2024-03-24T10:35:50.638820+0000","last_undegraded":"2024-03-24T10:35:50.638820+0000","last_fullsized":"2024-03-24T10:35:50.638820+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:12.007000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625329+0000","last_change":"2024-03-24T10:35:50.625329+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625329+0000","last_undegraded":"2024-03-24T10:35:50.625329+0000","last_fullsized":"2024-03-24T10:35:50.625329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638326+0000","last_change":"2024-03-24T10:35:50.632990+0000","last_active":"2024-03-24T10:35:50.638326+0000","last_peered":"2024-03-24T10:35:50.638326+0000","last_clean":"2024-03-24T10:35:50.638326+0000","last_became_active":"2024-03-24T10:35:50.632635+0000","last_became_peered":"2024-03-24T10:35:50.632635+0000","last_unstale":"2024-03-24T10:35:50.638326+0000","last_undegraded":"2024-03-24T10:35:50.638326+0000","last_fullsized":"2024-03-24T10:35:50.638326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:47.712468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623984+0000","last_change":"2024-03-24T10:35:50.623984+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623984+0000","last_undegraded":"2024-03-24T10:35:50.623984+0000","last_fullsized":"2024-03-24T10:35:50.623984+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639222+0000","last_change":"2024-03-24T10:35:50.638378+0000","last_active":"2024-03-24T10:35:50.639222+0000","last_peered":"2024-03-24T10:35:50.639222+0000","last_clean":"2024-03-24T10:35:50.639222+0000","last_became_active":"2024-03-24T10:35:50.638077+0000","last_became_peered":"2024-03-24T10:35:50.638077+0000","last_unstale":"2024-03-24T10:35:50.639222+0000","last_undegraded":"2024-03-24T10:35:50.639222+0000","last_fullsized":"2024-03-24T10:35:50.639222+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:48:48.966513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621207+0000","last_change":"2024-03-24T10:35:50.621207+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621207+0000","last_undegraded":"2024-03-24T10:35:50.621207+0000","last_fullsized":"2024-03-24T10:35:50.621207+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638629+0000","last_change":"2024-03-24T10:35:50.636470+0000","last_active":"2024-03-24T10:35:50.638629+0000","last_peered":"2024-03-24T10:35:50.638629+0000","last_clean":"2024-03-24T10:35:50.638629+0000","last_became_active":"2024-03-24T10:35:50.636139+0000","last_became_peered":"2024-03-24T10:35:50.636139+0000","last_unstale":"2024-03-24T10:35:50.638629+0000","last_undegraded":"2024-03-24T10:35:50.638629+0000","last_fullsized":"2024-03-24T10:35:50.638629+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:55.032277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624799+0000","last_change":"2024-03-24T10:35:50.624799+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624799+0000","last_undegraded":"2024-03-24T10:35:50.624799+0000","last_fullsized":"2024-03-24T10:35:50.624799+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637837+0000","last_change":"2024-03-24T10:35:50.635522+0000","last_active":"2024-03-24T10:35:50.637837+0000","last_peered":"2024-03-24T10:35:50.637837+0000","last_clean":"2024-03-24T10:35:50.637837+0000","last_became_active":"2024-03-24T10:35:50.635036+0000","last_became_peered":"2024-03-24T10:35:50.635036+0000","last_unstale":"2024-03-24T10:35:50.637837+0000","last_undegraded":"2024-03-24T10:35:50.637837+0000","last_fullsized":"2024-03-24T10:35:50.637837+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:55.117568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621406+0000","last_change":"2024-03-24T10:35:50.621406+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621406+0000","last_undegraded":"2024-03-24T10:35:50.621406+0000","last_fullsized":"2024-03-24T10:35:50.621406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638491+0000","last_change":"2024-03-24T10:35:50.636110+0000","last_active":"2024-03-24T10:35:50.638491+0000","last_peered":"2024-03-24T10:35:50.638491+0000","last_clean":"2024-03-24T10:35:50.638491+0000","last_became_active":"2024-03-24T10:35:50.635804+0000","last_became_peered":"2024-03-24T10:35:50.635804+0000","last_unstale":"2024-03-24T10:35:50.638491+0000","last_undegraded":"2024-03-24T10:35:50.638491+0000","last_fullsized":"2024-03-24T10:35:50.638491+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:26.730054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623498+0000","last_change":"2024-03-24T10:35:50.623498+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623498+0000","last_undegraded":"2024-03-24T10:35:50.623498+0000","last_fullsized":"2024-03-24T10:35:50.623498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634675+0000","last_change":"2024-03-24T10:35:50.630099+0000","last_active":"2024-03-24T10:35:50.634675+0000","last_peered":"2024-03-24T10:35:50.634675+0000","last_clean":"2024-03-24T10:35:50.634675+0000","last_became_active":"2024-03-24T10:35:50.629665+0000","last_became_peered":"2024-03-24T10:35:50.629665+0000","last_unstale":"2024-03-24T10:35:50.634675+0000","last_undegraded":"2024-03-24T10:35:50.634675+0000","last_fullsized":"2024-03-24T10:35:50.634675+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:10.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622768+0000","last_change":"2024-03-24T10:35:50.622768+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622768+0000","last_undegraded":"2024-03-24T10:35:50.622768+0000","last_fullsized":"2024-03-24T10:35:50.622768+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639380+0000","last_change":"2024-03-24T10:35:50.638305+0000","last_active":"2024-03-24T10:35:50.639380+0000","last_peered":"2024-03-24T10:35:50.639380+0000","last_clean":"2024-03-24T10:35:50.639380+0000","last_became_active":"2024-03-24T10:35:50.637768+0000","last_became_peered":"2024-03-24T10:35:50.637768+0000","last_unstale":"2024-03-24T10:35:50.639380+0000","last_undegraded":"2024-03-24T10:35:50.639380+0000","last_fullsized":"2024-03-24T10:35:50.639380+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:33:50.611475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623833+0000","last_change":"2024-03-24T10:35:50.623833+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623833+0000","last_undegraded":"2024-03-24T10:35:50.623833+0000","last_fullsized":"2024-03-24T10:35:50.623833+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634930+0000","last_change":"2024-03-24T10:35:50.631873+0000","last_active":"2024-03-24T10:35:50.634930+0000","last_peered":"2024-03-24T10:35:50.634930+0000","last_clean":"2024-03-24T10:35:50.634930+0000","last_became_active":"2024-03-24T10:35:50.631567+0000","last_became_peered":"2024-03-24T10:35:50.631567+0000","last_unstale":"2024-03-24T10:35:50.634930+0000","last_undegraded":"2024-03-24T10:35:50.634930+0000","last_fullsized":"2024-03-24T10:35:50.634930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:29.632034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623884+0000","last_change":"2024-03-24T10:35:50.623884+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623884+0000","last_undegraded":"2024-03-24T10:35:50.623884+0000","last_fullsized":"2024-03-24T10:35:50.623884+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632135+0000","last_change":"2024-03-24T10:35:50.626864+0000","last_active":"2024-03-24T10:35:50.632135+0000","last_peered":"2024-03-24T10:35:50.632135+0000","last_clean":"2024-03-24T10:35:50.632135+0000","last_became_active":"2024-03-24T10:35:50.626556+0000","last_became_peered":"2024-03-24T10:35:50.626556+0000","last_unstale":"2024-03-24T10:35:50.632135+0000","last_undegraded":"2024-03-24T10:35:50.632135+0000","last_fullsized":"2024-03-24T10:35:50.632135+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:40.363003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624072+0000","last_change":"2024-03-24T10:35:50.624072+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624072+0000","last_undegraded":"2024-03-24T10:35:50.624072+0000","last_fullsized":"2024-03-24T10:35:50.624072+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638533+0000","last_change":"2024-03-24T10:35:50.637003+0000","last_active":"2024-03-24T10:35:50.638533+0000","last_peered":"2024-03-24T10:35:50.638533+0000","last_clean":"2024-03-24T10:35:50.638533+0000","last_became_active":"2024-03-24T10:35:50.636716+0000","last_became_peered":"2024-03-24T10:35:50.636716+0000","last_unstale":"2024-03-24T10:35:50.638533+0000","last_undegraded":"2024-03-24T10:35:50.638533+0000","last_fullsized":"2024-03-24T10:35:50.638533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:13.161270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625445+0000","last_change":"2024-03-24T10:35:50.625445+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625445+0000","last_undegraded":"2024-03-24T10:35:50.625445+0000","last_fullsized":"2024-03-24T10:35:50.625445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638748+0000","last_change":"2024-03-24T10:35:50.637095+0000","last_active":"2024-03-24T10:35:50.638748+0000","last_peered":"2024-03-24T10:35:50.638748+0000","last_clean":"2024-03-24T10:35:50.638748+0000","last_became_active":"2024-03-24T10:35:50.636717+0000","last_became_peered":"2024-03-24T10:35:50.636717+0000","last_unstale":"2024-03-24T10:35:50.638748+0000","last_undegraded":"2024-03-24T10:35:50.638748+0000","last_fullsized":"2024-03-24T10:35:50.638748+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.035067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625235+0000","last_change":"2024-03-24T10:35:50.625235+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625235+0000","last_undegraded":"2024-03-24T10:35:50.625235+0000","last_fullsized":"2024-03-24T10:35:50.625235+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638235+0000","last_change":"2024-03-24T10:35:50.637013+0000","last_active":"2024-03-24T10:35:50.638235+0000","last_peered":"2024-03-24T10:35:50.638235+0000","last_clean":"2024-03-24T10:35:50.638235+0000","last_became_active":"2024-03-24T10:35:50.636281+0000","last_became_peered":"2024-03-24T10:35:50.636281+0000","last_unstale":"2024-03-24T10:35:50.638235+0000","last_undegraded":"2024-03-24T10:35:50.638235+0000","last_fullsized":"2024-03-24T10:35:50.638235+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:40.428655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.629902+0000","last_change":"2024-03-24T10:35:50.629902+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.629902+0000","last_undegraded":"2024-03-24T10:35:50.629902+0000","last_fullsized":"2024-03-24T10:35:50.629902+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.630082+0000","last_change":"2024-03-24T10:35:50.626837+0000","last_active":"2024-03-24T10:35:50.630082+0000","last_peered":"2024-03-24T10:35:50.630082+0000","last_clean":"2024-03-24T10:35:50.630082+0000","last_became_active":"2024-03-24T10:35:50.626538+0000","last_became_peered":"2024-03-24T10:35:50.626538+0000","last_unstale":"2024-03-24T10:35:50.630082+0000","last_undegraded":"2024-03-24T10:35:50.630082+0000","last_fullsized":"2024-03-24T10:35:50.630082+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:55.782184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619769+0000","last_change":"2024-03-24T10:35:50.619769+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619769+0000","last_undegraded":"2024-03-24T10:35:50.619769+0000","last_fullsized":"2024-03-24T10:35:50.619769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639683+0000","last_change":"2024-03-24T10:35:50.638452+0000","last_active":"2024-03-24T10:35:50.639683+0000","last_peered":"2024-03-24T10:35:50.639683+0000","last_clean":"2024-03-24T10:35:50.639683+0000","last_became_active":"2024-03-24T10:35:50.636353+0000","last_became_peered":"2024-03-24T10:35:50.636353+0000","last_unstale":"2024-03-24T10:35:50.639683+0000","last_undegraded":"2024-03-24T10:35:50.639683+0000","last_fullsized":"2024-03-24T10:35:50.639683+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:20.996736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618403+0000","last_change":"2024-03-24T10:35:50.618403+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618403+0000","last_undegraded":"2024-03-24T10:35:50.618403+0000","last_fullsized":"2024-03-24T10:35:50.618403+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.629210+0000","last_change":"2024-03-24T10:35:50.628002+0000","last_active":"2024-03-24T10:35:50.629210+0000","last_peered":"2024-03-24T10:35:50.629210+0000","last_clean":"2024-03-24T10:35:50.629210+0000","last_became_active":"2024-03-24T10:35:50.627615+0000","last_became_peered":"2024-03-24T10:35:50.627615+0000","last_unstale":"2024-03-24T10:35:50.629210+0000","last_undegraded":"2024-03-24T10:35:50.629210+0000","last_fullsized":"2024-03-24T10:35:50.629210+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:04:01.756032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624501+0000","last_change":"2024-03-24T10:35:50.624501+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624501+0000","last_undegraded":"2024-03-24T10:35:50.624501+0000","last_fullsized":"2024-03-24T10:35:50.624501+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635488+0000","last_change":"2024-03-24T10:35:50.632552+0000","last_active":"2024-03-24T10:35:50.635488+0000","last_peered":"2024-03-24T10:35:50.635488+0000","last_clean":"2024-03-24T10:35:50.635488+0000","last_became_active":"2024-03-24T10:35:50.632243+0000","last_became_peered":"2024-03-24T10:35:50.632243+0000","last_unstale":"2024-03-24T10:35:50.635488+0000","last_undegraded":"2024-03-24T10:35:50.635488+0000","last_fullsized":"2024-03-24T10:35:50.635488+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:44.471627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618657+0000","last_change":"2024-03-24T10:35:50.618657+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618657+0000","last_undegraded":"2024-03-24T10:35:50.618657+0000","last_fullsized":"2024-03-24T10:35:50.618657+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637051+0000","last_change":"2024-03-24T10:35:50.632996+0000","last_active":"2024-03-24T10:35:50.637051+0000","last_peered":"2024-03-24T10:35:50.637051+0000","last_clean":"2024-03-24T10:35:50.637051+0000","last_became_active":"2024-03-24T10:35:50.632669+0000","last_became_peered":"2024-03-24T10:35:50.632669+0000","last_unstale":"2024-03-24T10:35:50.637051+0000","last_undegraded":"2024-03-24T10:35:50.637051+0000","last_fullsized":"2024-03-24T10:35:50.637051+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:45:55.867715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623651+0000","last_change":"2024-03-24T10:35:50.623651+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623651+0000","last_undegraded":"2024-03-24T10:35:50.623651+0000","last_fullsized":"2024-03-24T10:35:50.623651+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.633750+0000","last_change":"2024-03-24T10:35:50.631866+0000","last_active":"2024-03-24T10:35:50.633750+0000","last_peered":"2024-03-24T10:35:50.633750+0000","last_clean":"2024-03-24T10:35:50.633750+0000","last_became_active":"2024-03-24T10:35:50.631578+0000","last_became_peered":"2024-03-24T10:35:50.631578+0000","last_unstale":"2024-03-24T10:35:50.633750+0000","last_undegraded":"2024-03-24T10:35:50.633750+0000","last_fullsized":"2024-03-24T10:35:50.633750+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:57.449362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.628548+0000","last_change":"2024-03-24T10:35:50.628548+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.628548+0000","last_undegraded":"2024-03-24T10:35:50.628548+0000","last_fullsized":"2024-03-24T10:35:50.628548+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632090+0000","last_change":"2024-03-24T10:35:50.629954+0000","last_active":"2024-03-24T10:35:50.632090+0000","last_peered":"2024-03-24T10:35:50.632090+0000","last_clean":"2024-03-24T10:35:50.632090+0000","last_became_active":"2024-03-24T10:35:50.629658+0000","last_became_peered":"2024-03-24T10:35:50.629658+0000","last_unstale":"2024-03-24T10:35:50.632090+0000","last_undegraded":"2024-03-24T10:35:50.632090+0000","last_fullsized":"2024-03-24T10:35:50.632090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:41.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626260+0000","last_change":"2024-03-24T10:35:50.626260+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626260+0000","last_undegraded":"2024-03-24T10:35:50.626260+0000","last_fullsized":"2024-03-24T10:35:50.626260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637838+0000","last_change":"2024-03-24T10:35:50.635109+0000","last_active":"2024-03-24T10:35:50.637838+0000","last_peered":"2024-03-24T10:35:50.637838+0000","last_clean":"2024-03-24T10:35:50.637838+0000","last_became_active":"2024-03-24T10:35:50.634804+0000","last_became_peered":"2024-03-24T10:35:50.634804+0000","last_unstale":"2024-03-24T10:35:50.637838+0000","last_undegraded":"2024-03-24T10:35:50.637838+0000","last_fullsized":"2024-03-24T10:35:50.637838+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:41.493981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618384+0000","last_change":"2024-03-24T10:35:50.618384+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618384+0000","last_undegraded":"2024-03-24T10:35:50.618384+0000","last_fullsized":"2024-03-24T10:35:50.618384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632261+0000","last_change":"2024-03-24T10:35:50.630807+0000","last_active":"2024-03-24T10:35:50.632261+0000","last_peered":"2024-03-24T10:35:50.632261+0000","last_clean":"2024-03-24T10:35:50.632261+0000","last_became_active":"2024-03-24T10:35:50.630512+0000","last_became_peered":"2024-03-24T10:35:50.630512+0000","last_unstale":"2024-03-24T10:35:50.632261+0000","last_undegraded":"2024-03-24T10:35:50.632261+0000","last_fullsized":"2024-03-24T10:35:50.632261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:55.432206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623104+0000","last_change":"2024-03-24T10:35:50.623104+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623104+0000","last_undegraded":"2024-03-24T10:35:50.623104+0000","last_fullsized":"2024-03-24T10:35:50.623104+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639574+0000","last_change":"2024-03-24T10:35:50.638566+0000","last_active":"2024-03-24T10:35:50.639574+0000","last_peered":"2024-03-24T10:35:50.639574+0000","last_clean":"2024-03-24T10:35:50.639574+0000","last_became_active":"2024-03-24T10:35:50.638280+0000","last_became_peered":"2024-03-24T10:35:50.638280+0000","last_unstale":"2024-03-24T10:35:50.639574+0000","last_undegraded":"2024-03-24T10:35:50.639574+0000","last_fullsized":"2024-03-24T10:35:50.639574+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:29.065867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618732+0000","last_change":"2024-03-24T10:35:50.618732+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618732+0000","last_undegraded":"2024-03-24T10:35:50.618732+0000","last_fullsized":"2024-03-24T10:35:50.618732+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634647+0000","last_change":"2024-03-24T10:35:50.631008+0000","last_active":"2024-03-24T10:35:50.634647+0000","last_peered":"2024-03-24T10:35:50.634647+0000","last_clean":"2024-03-24T10:35:50.634647+0000","last_became_active":"2024-03-24T10:35:50.630722+0000","last_became_peered":"2024-03-24T10:35:50.630722+0000","last_unstale":"2024-03-24T10:35:50.634647+0000","last_undegraded":"2024-03-24T10:35:50.634647+0000","last_fullsized":"2024-03-24T10:35:50.634647+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:21.376175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624509+0000","last_change":"2024-03-24T10:35:50.624509+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624509+0000","last_undegraded":"2024-03-24T10:35:50.624509+0000","last_fullsized":"2024-03-24T10:35:50.624509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638327+0000","last_change":"2024-03-24T10:35:50.636330+0000","last_active":"2024-03-24T10:35:50.638327+0000","last_peered":"2024-03-24T10:35:50.638327+0000","last_clean":"2024-03-24T10:35:50.638327+0000","last_became_active":"2024-03-24T10:35:50.636035+0000","last_became_peered":"2024-03-24T10:35:50.636035+0000","last_unstale":"2024-03-24T10:35:50.638327+0000","last_undegraded":"2024-03-24T10:35:50.638327+0000","last_fullsized":"2024-03-24T10:35:50.638327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:54.961171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624517+0000","last_change":"2024-03-24T10:35:50.624517+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624517+0000","last_undegraded":"2024-03-24T10:35:50.624517+0000","last_fullsized":"2024-03-24T10:35:50.624517+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634211+0000","last_change":"2024-03-24T10:35:50.628958+0000","last_active":"2024-03-24T10:35:50.634211+0000","last_peered":"2024-03-24T10:35:50.634211+0000","last_clean":"2024-03-24T10:35:50.634211+0000","last_became_active":"2024-03-24T10:35:50.628661+0000","last_became_peered":"2024-03-24T10:35:50.628661+0000","last_unstale":"2024-03-24T10:35:50.634211+0000","last_undegraded":"2024-03-24T10:35:50.634211+0000","last_fullsized":"2024-03-24T10:35:50.634211+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:01.210812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623387+0000","last_change":"2024-03-24T10:35:50.623387+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623387+0000","last_undegraded":"2024-03-24T10:35:50.623387+0000","last_fullsized":"2024-03-24T10:35:50.623387+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632067+0000","last_change":"2024-03-24T10:35:50.627406+0000","last_active":"2024-03-24T10:35:50.632067+0000","last_peered":"2024-03-24T10:35:50.632067+0000","last_clean":"2024-03-24T10:35:50.632067+0000","last_became_active":"2024-03-24T10:35:50.627116+0000","last_became_peered":"2024-03-24T10:35:50.627116+0000","last_unstale":"2024-03-24T10:35:50.632067+0000","last_undegraded":"2024-03-24T10:35:50.632067+0000","last_fullsized":"2024-03-24T10:35:50.632067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:44.632873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630039+0000","last_change":"2024-03-24T10:35:50.630039+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630039+0000","last_undegraded":"2024-03-24T10:35:50.630039+0000","last_fullsized":"2024-03-24T10:35:50.630039+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638602+0000","last_change":"2024-03-24T10:35:50.635584+0000","last_active":"2024-03-24T10:35:50.638602+0000","last_peered":"2024-03-24T10:35:50.638602+0000","last_clean":"2024-03-24T10:35:50.638602+0000","last_became_active":"2024-03-24T10:35:50.635280+0000","last_became_peered":"2024-03-24T10:35:50.635280+0000","last_unstale":"2024-03-24T10:35:50.638602+0000","last_undegraded":"2024-03-24T10:35:50.638602+0000","last_fullsized":"2024-03-24T10:35:50.638602+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:27:14.345670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622626+0000","last_change":"2024-03-24T10:35:50.622626+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622626+0000","last_undegraded":"2024-03-24T10:35:50.622626+0000","last_fullsized":"2024-03-24T10:35:50.622626+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635400+0000","last_change":"2024-03-24T10:35:50.631769+0000","last_active":"2024-03-24T10:35:50.635400+0000","last_peered":"2024-03-24T10:35:50.635400+0000","last_clean":"2024-03-24T10:35:50.635400+0000","last_became_active":"2024-03-24T10:35:50.631453+0000","last_became_peered":"2024-03-24T10:35:50.631453+0000","last_unstale":"2024-03-24T10:35:50.635400+0000","last_undegraded":"2024-03-24T10:35:50.635400+0000","last_fullsized":"2024-03-24T10:35:50.635400+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:11.009937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626484+0000","last_change":"2024-03-24T10:35:50.626484+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626484+0000","last_undegraded":"2024-03-24T10:35:50.626484+0000","last_fullsized":"2024-03-24T10:35:50.626484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638457+0000","last_change":"2024-03-24T10:35:50.636812+0000","last_active":"2024-03-24T10:35:50.638457+0000","last_peered":"2024-03-24T10:35:50.638457+0000","last_clean":"2024-03-24T10:35:50.638457+0000","last_became_active":"2024-03-24T10:35:50.636520+0000","last_became_peered":"2024-03-24T10:35:50.636520+0000","last_unstale":"2024-03-24T10:35:50.638457+0000","last_undegraded":"2024-03-24T10:35:50.638457+0000","last_fullsized":"2024-03-24T10:35:50.638457+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:16.151719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.622920+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:50.622920+0000","last_peered":"2024-03-24T10:35:50.622920+0000","last_clean":"2024-03-24T10:35:50.622920+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:50.622920+0000","last_undegraded":"2024-03-24T10:35:50.622920+0000","last_fullsized":"2024-03-24T10:35:50.622920+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":227616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:35:54.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:35:54.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.832+0000 7f0170f9a640 1 Processor -- start 2024-03-24T10:35:54.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.832+0000 7f0170f9a640 1 -- start start 2024-03-24T10:35:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f0170f9a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016c0d1000 0x7f016c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f0170f9a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f016c15e770 con 0x7f016c15e390 2024-03-24T10:35:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f0170f9a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f016c0d4e40 con 0x7f016c0d2bb0 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f0170f9a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f016c0b3190 con 0x7f016c0d1000 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016c0d1000 0x7f016c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016c0d1000 0x7f016c0b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38596/0 (socket says 172.21.15.1:38596) 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f016c15e390 0x7f016c0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49918/0 (socket says 172.21.15.1:49918) 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 -- 172.21.15.1:0/3862339836 learned_addr learned my addr 172.21.15.1:0/3862339836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 -- 172.21.15.1:0/3862339836 >> v1:172.21.15.3:6789/0 conn(0x7f016c0d2bb0 legacy=0x7f016c0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 -- 172.21.15.1:0/3862339836 >> v1:172.21.15.1:6789/0 conn(0x7f016c15e390 legacy=0x7f016c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016ad9d640 1 -- 172.21.15.1:0/3862339836 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016c0d1e40 con 0x7f016c0d1000 2024-03-24T10:35:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.833+0000 7f016959a640 1 -- 172.21.15.1:0/3862339836 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181965645 0 0) 0x7f015c001110 con 0x7f016c15e390 2024-03-24T10:35:54.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.834+0000 7f016ad9d640 1 --2- 172.21.15.1:0/3862339836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b42a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0164009e50 tx=0x7f016402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41debb6b044b3818 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.834+0000 7f016959a640 1 -- 172.21.15.1:0/3862339836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016402ca80 con 0x7f016c0d1000 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.834+0000 7f016959a640 1 -- 172.21.15.1:0/3862339836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016402cbe0 con 0x7f016c0d1000 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.834+0000 7f016959a640 1 -- 172.21.15.1:0/3862339836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640343f0 con 0x7f016c0d1000 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 -- 172.21.15.1:0/3862339836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 msgr2=0x7f016c0b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 --2- 172.21.15.1:0/3862339836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b42a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0164009e50 tx=0x7f016402bee0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 -- 172.21.15.1:0/3862339836 shutdown_connections 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 --2- 172.21.15.1:0/3862339836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b42a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 -- 172.21.15.1:0/3862339836 >> 172.21.15.1:0/3862339836 conn(0x7f016c0d3ec0 msgr2=0x7f016c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 -- 172.21.15.1:0/3862339836 shutdown_connections 2024-03-24T10:35:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.835+0000 7f0170f9a640 1 -- 172.21.15.1:0/3862339836 wait complete. 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.836+0000 7f0170f9a640 1 Processor -- start 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.836+0000 7f0170f9a640 1 -- start start 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 0x7f016c09fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 0x7f016c09dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f016c0a1540 con 0x7f016c0d1000 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f016c0b94b0 con 0x7f016c0d2bb0 2024-03-24T10:35:54.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0170f9a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f016c0a1960 con 0x7f016c15e390 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38598/0 (socket says 172.21.15.1:38598) 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 -- 172.21.15.1:0/397435890 learned_addr learned my addr 172.21.15.1:0/397435890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0169d9b640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 0x7f016c09fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016ad9d640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 0x7f016c09dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 -- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 msgr2=0x7f016c09dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 0x7f016c09dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 -- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 msgr2=0x7f016c09fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 0x7f016c09fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016a59c640 1 -- 172.21.15.1:0/397435890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0164009b30 con 0x7f016c0d1000 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f0169d9b640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 0x7f016c09fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.837+0000 7f016ad9d640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 0x7f016c09dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f016a59c640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f015c00c420 tx=0x7f015c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c0074e0 con 0x7f016c0d1000 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015c007640 con 0x7f016c0d1000 2024-03-24T10:35:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c004a00 con 0x7f016c0d1000 2024-03-24T10:35:54.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016c0a08c0 con 0x7f016c0d1000 2024-03-24T10:35:54.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.838+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016c0a0e80 con 0x7f016c0d1000 2024-03-24T10:35:54.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.839+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016c0b9dd0 con 0x7f016c0d1000 2024-03-24T10:35:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.839+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f015c004b60 con 0x7f016c0d1000 2024-03-24T10:35:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.840+0000 7f015affd640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 0x7f014c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.840+0000 7f0169d9b640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 0x7f014c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.840+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f015c089e40 con 0x7f016c0d1000 2024-03-24T10:35:54.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.840+0000 7f0169d9b640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 0x7f014c06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0160007fd0 tx=0x7f0160009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:54.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:54.844+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f015c054860 con 0x7f016c0d1000 2024-03-24T10:35:55.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.020+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f016c0b3320 con 0x7f014c06b810 2024-03-24T10:35:55.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.026+0000 7f015affd640 1 -- 172.21.15.1:0/397435890 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246641 (secure 0 0 0) 0x7f01600013e0 con 0x7f014c06b810 2024-03-24T10:35:55.029 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:55.030 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 msgr2=0x7f014c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 0x7f014c06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0160007fd0 tx=0x7f0160009040 comp rx=0 tx=0).stop 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 msgr2=0x7f016c0b8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f015c00c420 tx=0x7f015c00c8e0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 shutdown_connections 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014c06b810 0x7f014c06dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016c15e390 0x7f016c09dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016c0d2bb0 0x7f016c09fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 --2- 172.21.15.1:0/397435890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016c0d1000 0x7f016c0b8f70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 >> 172.21.15.1:0/397435890 conn(0x7f016c0d3ec0 msgr2=0x7f016c0d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 shutdown_connections 2024-03-24T10:35:55.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.029+0000 7f0170f9a640 1 -- 172.21.15.1:0/397435890 wait complete. 2024-03-24T10:35:55.040 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-03-24T10:35:54.539396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238424,"kb_used_data":5596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736296,"statfs":{"total":773094113280,"available":772849967104,"internally_reserved":0,"allocated":5730304,"data_stored":3045589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931065"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619572+0000","last_change":"2024-03-24T10:35:50.619572+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619572+0000","last_undegraded":"2024-03-24T10:35:50.619572+0000","last_fullsized":"2024-03-24T10:35:50.619572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622464+0000","last_change":"2024-03-24T10:35:50.622464+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622464+0000","last_undegraded":"2024-03-24T10:35:50.622464+0000","last_fullsized":"2024-03-24T10:35:50.622464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627689+0000","last_change":"2024-03-24T10:35:50.627689+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627689+0000","last_undegraded":"2024-03-24T10:35:50.627689+0000","last_fullsized":"2024-03-24T10:35:50.627689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621316+0000","last_change":"2024-03-24T10:35:50.621316+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621316+0000","last_undegraded":"2024-03-24T10:35:50.621316+0000","last_fullsized":"2024-03-24T10:35:50.621316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624565+0000","last_change":"2024-03-24T10:35:50.624565+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624565+0000","last_undegraded":"2024-03-24T10:35:50.624565+0000","last_fullsized":"2024-03-24T10:35:50.624565+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619946+0000","last_change":"2024-03-24T10:35:50.619946+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619946+0000","last_undegraded":"2024-03-24T10:35:50.619946+0000","last_fullsized":"2024-03-24T10:35:50.619946+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624479+0000","last_change":"2024-03-24T10:35:50.624479+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624479+0000","last_undegraded":"2024-03-24T10:35:50.624479+0000","last_fullsized":"2024-03-24T10:35:50.624479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624154+0000","last_change":"2024-03-24T10:35:50.624154+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624154+0000","last_undegraded":"2024-03-24T10:35:50.624154+0000","last_fullsized":"2024-03-24T10:35:50.624154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623889+0000","last_change":"2024-03-24T10:35:50.623889+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623889+0000","last_undegraded":"2024-03-24T10:35:50.623889+0000","last_fullsized":"2024-03-24T10:35:50.623889+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619433+0000","last_change":"2024-03-24T10:35:50.619433+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619433+0000","last_undegraded":"2024-03-24T10:35:50.619433+0000","last_fullsized":"2024-03-24T10:35:50.619433+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619034+0000","last_change":"2024-03-24T10:35:50.619034+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619034+0000","last_undegraded":"2024-03-24T10:35:50.619034+0000","last_fullsized":"2024-03-24T10:35:50.619034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622027+0000","last_change":"2024-03-24T10:35:50.622027+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622027+0000","last_undegraded":"2024-03-24T10:35:50.622027+0000","last_fullsized":"2024-03-24T10:35:50.622027+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619363+0000","last_change":"2024-03-24T10:35:50.619363+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619363+0000","last_undegraded":"2024-03-24T10:35:50.619363+0000","last_fullsized":"2024-03-24T10:35:50.619363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624300+0000","last_change":"2024-03-24T10:35:50.624300+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624300+0000","last_undegraded":"2024-03-24T10:35:50.624300+0000","last_fullsized":"2024-03-24T10:35:50.624300+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621221+0000","last_change":"2024-03-24T10:35:50.621221+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621221+0000","last_undegraded":"2024-03-24T10:35:50.621221+0000","last_fullsized":"2024-03-24T10:35:50.621221+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618425+0000","last_change":"2024-03-24T10:35:50.618425+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618425+0000","last_undegraded":"2024-03-24T10:35:50.618425+0000","last_fullsized":"2024-03-24T10:35:50.618425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627699+0000","last_change":"2024-03-24T10:35:50.627699+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627699+0000","last_undegraded":"2024-03-24T10:35:50.627699+0000","last_fullsized":"2024-03-24T10:35:50.627699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622054+0000","last_change":"2024-03-24T10:35:50.622054+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622054+0000","last_undegraded":"2024-03-24T10:35:50.622054+0000","last_fullsized":"2024-03-24T10:35:50.622054+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627222+0000","last_change":"2024-03-24T10:35:50.627222+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627222+0000","last_undegraded":"2024-03-24T10:35:50.627222+0000","last_fullsized":"2024-03-24T10:35:50.627222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627419+0000","last_change":"2024-03-24T10:35:50.627419+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627419+0000","last_undegraded":"2024-03-24T10:35:50.627419+0000","last_fullsized":"2024-03-24T10:35:50.627419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625486+0000","last_change":"2024-03-24T10:35:50.625486+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625486+0000","last_undegraded":"2024-03-24T10:35:50.625486+0000","last_fullsized":"2024-03-24T10:35:50.625486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622677+0000","last_change":"2024-03-24T10:35:50.622677+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622677+0000","last_undegraded":"2024-03-24T10:35:50.622677+0000","last_fullsized":"2024-03-24T10:35:50.622677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618841+0000","last_change":"2024-03-24T10:35:50.618841+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618841+0000","last_undegraded":"2024-03-24T10:35:50.618841+0000","last_fullsized":"2024-03-24T10:35:50.618841+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622708+0000","last_change":"2024-03-24T10:35:50.622708+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622708+0000","last_undegraded":"2024-03-24T10:35:50.622708+0000","last_fullsized":"2024-03-24T10:35:50.622708+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625709+0000","last_change":"2024-03-24T10:35:50.625709+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625709+0000","last_undegraded":"2024-03-24T10:35:50.625709+0000","last_fullsized":"2024-03-24T10:35:50.625709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625691+0000","last_change":"2024-03-24T10:35:50.625691+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625691+0000","last_undegraded":"2024-03-24T10:35:50.625691+0000","last_fullsized":"2024-03-24T10:35:50.625691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618448+0000","last_change":"2024-03-24T10:35:50.618448+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618448+0000","last_undegraded":"2024-03-24T10:35:50.618448+0000","last_fullsized":"2024-03-24T10:35:50.618448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626365+0000","last_change":"2024-03-24T10:35:50.626365+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626365+0000","last_undegraded":"2024-03-24T10:35:50.626365+0000","last_fullsized":"2024-03-24T10:35:50.626365+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630750+0000","last_change":"2024-03-24T10:35:50.630750+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630750+0000","last_undegraded":"2024-03-24T10:35:50.630750+0000","last_fullsized":"2024-03-24T10:35:50.630750+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621203+0000","last_change":"2024-03-24T10:35:50.621203+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621203+0000","last_undegraded":"2024-03-24T10:35:50.621203+0000","last_fullsized":"2024-03-24T10:35:50.621203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623326+0000","last_change":"2024-03-24T10:35:50.623326+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623326+0000","last_undegraded":"2024-03-24T10:35:50.623326+0000","last_fullsized":"2024-03-24T10:35:50.623326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624597+0000","last_change":"2024-03-24T10:35:50.624597+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624597+0000","last_undegraded":"2024-03-24T10:35:50.624597+0000","last_fullsized":"2024-03-24T10:35:50.624597+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621815+0000","last_change":"2024-03-24T10:35:50.621815+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621815+0000","last_undegraded":"2024-03-24T10:35:50.621815+0000","last_fullsized":"2024-03-24T10:35:50.621815+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638164+0000","last_change":"2024-03-24T10:35:50.635274+0000","last_active":"2024-03-24T10:35:50.638164+0000","last_peered":"2024-03-24T10:35:50.638164+0000","last_clean":"2024-03-24T10:35:50.638164+0000","last_became_active":"2024-03-24T10:35:50.634934+0000","last_became_peered":"2024-03-24T10:35:50.634934+0000","last_unstale":"2024-03-24T10:35:50.638164+0000","last_undegraded":"2024-03-24T10:35:50.638164+0000","last_fullsized":"2024-03-24T10:35:50.638164+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:19.768286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622233+0000","last_change":"2024-03-24T10:35:50.622233+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622233+0000","last_undegraded":"2024-03-24T10:35:50.622233+0000","last_fullsized":"2024-03-24T10:35:50.622233+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634816+0000","last_change":"2024-03-24T10:35:50.633891+0000","last_active":"2024-03-24T10:35:50.634816+0000","last_peered":"2024-03-24T10:35:50.634816+0000","last_clean":"2024-03-24T10:35:50.634816+0000","last_became_active":"2024-03-24T10:35:50.633604+0000","last_became_peered":"2024-03-24T10:35:50.633604+0000","last_unstale":"2024-03-24T10:35:50.634816+0000","last_undegraded":"2024-03-24T10:35:50.634816+0000","last_fullsized":"2024-03-24T10:35:50.634816+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:45.059275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626163+0000","last_change":"2024-03-24T10:35:50.626163+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626163+0000","last_undegraded":"2024-03-24T10:35:50.626163+0000","last_fullsized":"2024-03-24T10:35:50.626163+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638820+0000","last_change":"2024-03-24T10:35:50.637492+0000","last_active":"2024-03-24T10:35:50.638820+0000","last_peered":"2024-03-24T10:35:50.638820+0000","last_clean":"2024-03-24T10:35:50.638820+0000","last_became_active":"2024-03-24T10:35:50.637225+0000","last_became_peered":"2024-03-24T10:35:50.637225+0000","last_unstale":"2024-03-24T10:35:50.638820+0000","last_undegraded":"2024-03-24T10:35:50.638820+0000","last_fullsized":"2024-03-24T10:35:50.638820+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:12.007000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625329+0000","last_change":"2024-03-24T10:35:50.625329+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625329+0000","last_undegraded":"2024-03-24T10:35:50.625329+0000","last_fullsized":"2024-03-24T10:35:50.625329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638326+0000","last_change":"2024-03-24T10:35:50.632990+0000","last_active":"2024-03-24T10:35:50.638326+0000","last_peered":"2024-03-24T10:35:50.638326+0000","last_clean":"2024-03-24T10:35:50.638326+0000","last_became_active":"2024-03-24T10:35:50.632635+0000","last_became_peered":"2024-03-24T10:35:50.632635+0000","last_unstale":"2024-03-24T10:35:50.638326+0000","last_undegraded":"2024-03-24T10:35:50.638326+0000","last_fullsized":"2024-03-24T10:35:50.638326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:47.712468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623984+0000","last_change":"2024-03-24T10:35:50.623984+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623984+0000","last_undegraded":"2024-03-24T10:35:50.623984+0000","last_fullsized":"2024-03-24T10:35:50.623984+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639222+0000","last_change":"2024-03-24T10:35:50.638378+0000","last_active":"2024-03-24T10:35:50.639222+0000","last_peered":"2024-03-24T10:35:50.639222+0000","last_clean":"2024-03-24T10:35:50.639222+0000","last_became_active":"2024-03-24T10:35:50.638077+0000","last_became_peered":"2024-03-24T10:35:50.638077+0000","last_unstale":"2024-03-24T10:35:50.639222+0000","last_undegraded":"2024-03-24T10:35:50.639222+0000","last_fullsized":"2024-03-24T10:35:50.639222+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:48:48.966513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621207+0000","last_change":"2024-03-24T10:35:50.621207+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621207+0000","last_undegraded":"2024-03-24T10:35:50.621207+0000","last_fullsized":"2024-03-24T10:35:50.621207+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638629+0000","last_change":"2024-03-24T10:35:50.636470+0000","last_active":"2024-03-24T10:35:50.638629+0000","last_peered":"2024-03-24T10:35:50.638629+0000","last_clean":"2024-03-24T10:35:50.638629+0000","last_became_active":"2024-03-24T10:35:50.636139+0000","last_became_peered":"2024-03-24T10:35:50.636139+0000","last_unstale":"2024-03-24T10:35:50.638629+0000","last_undegraded":"2024-03-24T10:35:50.638629+0000","last_fullsized":"2024-03-24T10:35:50.638629+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:55.032277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624799+0000","last_change":"2024-03-24T10:35:50.624799+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624799+0000","last_undegraded":"2024-03-24T10:35:50.624799+0000","last_fullsized":"2024-03-24T10:35:50.624799+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637837+0000","last_change":"2024-03-24T10:35:50.635522+0000","last_active":"2024-03-24T10:35:50.637837+0000","last_peered":"2024-03-24T10:35:50.637837+0000","last_clean":"2024-03-24T10:35:50.637837+0000","last_became_active":"2024-03-24T10:35:50.635036+0000","last_became_peered":"2024-03-24T10:35:50.635036+0000","last_unstale":"2024-03-24T10:35:50.637837+0000","last_undegraded":"2024-03-24T10:35:50.637837+0000","last_fullsized":"2024-03-24T10:35:50.637837+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:55.117568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621406+0000","last_change":"2024-03-24T10:35:50.621406+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621406+0000","last_undegraded":"2024-03-24T10:35:50.621406+0000","last_fullsized":"2024-03-24T10:35:50.621406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638491+0000","last_change":"2024-03-24T10:35:50.636110+0000","last_active":"2024-03-24T10:35:50.638491+0000","last_peered":"2024-03-24T10:35:50.638491+0000","last_clean":"2024-03-24T10:35:50.638491+0000","last_became_active":"2024-03-24T10:35:50.635804+0000","last_became_peered":"2024-03-24T10:35:50.635804+0000","last_unstale":"2024-03-24T10:35:50.638491+0000","last_undegraded":"2024-03-24T10:35:50.638491+0000","last_fullsized":"2024-03-24T10:35:50.638491+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:26.730054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623498+0000","last_change":"2024-03-24T10:35:50.623498+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623498+0000","last_undegraded":"2024-03-24T10:35:50.623498+0000","last_fullsized":"2024-03-24T10:35:50.623498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634675+0000","last_change":"2024-03-24T10:35:50.630099+0000","last_active":"2024-03-24T10:35:50.634675+0000","last_peered":"2024-03-24T10:35:50.634675+0000","last_clean":"2024-03-24T10:35:50.634675+0000","last_became_active":"2024-03-24T10:35:50.629665+0000","last_became_peered":"2024-03-24T10:35:50.629665+0000","last_unstale":"2024-03-24T10:35:50.634675+0000","last_undegraded":"2024-03-24T10:35:50.634675+0000","last_fullsized":"2024-03-24T10:35:50.634675+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:10.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622768+0000","last_change":"2024-03-24T10:35:50.622768+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622768+0000","last_undegraded":"2024-03-24T10:35:50.622768+0000","last_fullsized":"2024-03-24T10:35:50.622768+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639380+0000","last_change":"2024-03-24T10:35:50.638305+0000","last_active":"2024-03-24T10:35:50.639380+0000","last_peered":"2024-03-24T10:35:50.639380+0000","last_clean":"2024-03-24T10:35:50.639380+0000","last_became_active":"2024-03-24T10:35:50.637768+0000","last_became_peered":"2024-03-24T10:35:50.637768+0000","last_unstale":"2024-03-24T10:35:50.639380+0000","last_undegraded":"2024-03-24T10:35:50.639380+0000","last_fullsized":"2024-03-24T10:35:50.639380+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:33:50.611475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623833+0000","last_change":"2024-03-24T10:35:50.623833+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623833+0000","last_undegraded":"2024-03-24T10:35:50.623833+0000","last_fullsized":"2024-03-24T10:35:50.623833+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634930+0000","last_change":"2024-03-24T10:35:50.631873+0000","last_active":"2024-03-24T10:35:50.634930+0000","last_peered":"2024-03-24T10:35:50.634930+0000","last_clean":"2024-03-24T10:35:50.634930+0000","last_became_active":"2024-03-24T10:35:50.631567+0000","last_became_peered":"2024-03-24T10:35:50.631567+0000","last_unstale":"2024-03-24T10:35:50.634930+0000","last_undegraded":"2024-03-24T10:35:50.634930+0000","last_fullsized":"2024-03-24T10:35:50.634930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:29.632034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623884+0000","last_change":"2024-03-24T10:35:50.623884+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623884+0000","last_undegraded":"2024-03-24T10:35:50.623884+0000","last_fullsized":"2024-03-24T10:35:50.623884+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632135+0000","last_change":"2024-03-24T10:35:50.626864+0000","last_active":"2024-03-24T10:35:50.632135+0000","last_peered":"2024-03-24T10:35:50.632135+0000","last_clean":"2024-03-24T10:35:50.632135+0000","last_became_active":"2024-03-24T10:35:50.626556+0000","last_became_peered":"2024-03-24T10:35:50.626556+0000","last_unstale":"2024-03-24T10:35:50.632135+0000","last_undegraded":"2024-03-24T10:35:50.632135+0000","last_fullsized":"2024-03-24T10:35:50.632135+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:40.363003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624072+0000","last_change":"2024-03-24T10:35:50.624072+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624072+0000","last_undegraded":"2024-03-24T10:35:50.624072+0000","last_fullsized":"2024-03-24T10:35:50.624072+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638533+0000","last_change":"2024-03-24T10:35:50.637003+0000","last_active":"2024-03-24T10:35:50.638533+0000","last_peered":"2024-03-24T10:35:50.638533+0000","last_clean":"2024-03-24T10:35:50.638533+0000","last_became_active":"2024-03-24T10:35:50.636716+0000","last_became_peered":"2024-03-24T10:35:50.636716+0000","last_unstale":"2024-03-24T10:35:50.638533+0000","last_undegraded":"2024-03-24T10:35:50.638533+0000","last_fullsized":"2024-03-24T10:35:50.638533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:13.161270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625445+0000","last_change":"2024-03-24T10:35:50.625445+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625445+0000","last_undegraded":"2024-03-24T10:35:50.625445+0000","last_fullsized":"2024-03-24T10:35:50.625445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638748+0000","last_change":"2024-03-24T10:35:50.637095+0000","last_active":"2024-03-24T10:35:50.638748+0000","last_peered":"2024-03-24T10:35:50.638748+0000","last_clean":"2024-03-24T10:35:50.638748+0000","last_became_active":"2024-03-24T10:35:50.636717+0000","last_became_peered":"2024-03-24T10:35:50.636717+0000","last_unstale":"2024-03-24T10:35:50.638748+0000","last_undegraded":"2024-03-24T10:35:50.638748+0000","last_fullsized":"2024-03-24T10:35:50.638748+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.035067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625235+0000","last_change":"2024-03-24T10:35:50.625235+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625235+0000","last_undegraded":"2024-03-24T10:35:50.625235+0000","last_fullsized":"2024-03-24T10:35:50.625235+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638235+0000","last_change":"2024-03-24T10:35:50.637013+0000","last_active":"2024-03-24T10:35:50.638235+0000","last_peered":"2024-03-24T10:35:50.638235+0000","last_clean":"2024-03-24T10:35:50.638235+0000","last_became_active":"2024-03-24T10:35:50.636281+0000","last_became_peered":"2024-03-24T10:35:50.636281+0000","last_unstale":"2024-03-24T10:35:50.638235+0000","last_undegraded":"2024-03-24T10:35:50.638235+0000","last_fullsized":"2024-03-24T10:35:50.638235+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:40.428655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.629902+0000","last_change":"2024-03-24T10:35:50.629902+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.629902+0000","last_undegraded":"2024-03-24T10:35:50.629902+0000","last_fullsized":"2024-03-24T10:35:50.629902+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.630082+0000","last_change":"2024-03-24T10:35:50.626837+0000","last_active":"2024-03-24T10:35:50.630082+0000","last_peered":"2024-03-24T10:35:50.630082+0000","last_clean":"2024-03-24T10:35:50.630082+0000","last_became_active":"2024-03-24T10:35:50.626538+0000","last_became_peered":"2024-03-24T10:35:50.626538+0000","last_unstale":"2024-03-24T10:35:50.630082+0000","last_undegraded":"2024-03-24T10:35:50.630082+0000","last_fullsized":"2024-03-24T10:35:50.630082+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:55.782184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619769+0000","last_change":"2024-03-24T10:35:50.619769+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619769+0000","last_undegraded":"2024-03-24T10:35:50.619769+0000","last_fullsized":"2024-03-24T10:35:50.619769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639683+0000","last_change":"2024-03-24T10:35:50.638452+0000","last_active":"2024-03-24T10:35:50.639683+0000","last_peered":"2024-03-24T10:35:50.639683+0000","last_clean":"2024-03-24T10:35:50.639683+0000","last_became_active":"2024-03-24T10:35:50.636353+0000","last_became_peered":"2024-03-24T10:35:50.636353+0000","last_unstale":"2024-03-24T10:35:50.639683+0000","last_undegraded":"2024-03-24T10:35:50.639683+0000","last_fullsized":"2024-03-24T10:35:50.639683+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:20.996736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618403+0000","last_change":"2024-03-24T10:35:50.618403+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618403+0000","last_undegraded":"2024-03-24T10:35:50.618403+0000","last_fullsized":"2024-03-24T10:35:50.618403+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.629210+0000","last_change":"2024-03-24T10:35:50.628002+0000","last_active":"2024-03-24T10:35:50.629210+0000","last_peered":"2024-03-24T10:35:50.629210+0000","last_clean":"2024-03-24T10:35:50.629210+0000","last_became_active":"2024-03-24T10:35:50.627615+0000","last_became_peered":"2024-03-24T10:35:50.627615+0000","last_unstale":"2024-03-24T10:35:50.629210+0000","last_undegraded":"2024-03-24T10:35:50.629210+0000","last_fullsized":"2024-03-24T10:35:50.629210+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:04:01.756032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624501+0000","last_change":"2024-03-24T10:35:50.624501+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624501+0000","last_undegraded":"2024-03-24T10:35:50.624501+0000","last_fullsized":"2024-03-24T10:35:50.624501+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635488+0000","last_change":"2024-03-24T10:35:50.632552+0000","last_active":"2024-03-24T10:35:50.635488+0000","last_peered":"2024-03-24T10:35:50.635488+0000","last_clean":"2024-03-24T10:35:50.635488+0000","last_became_active":"2024-03-24T10:35:50.632243+0000","last_became_peered":"2024-03-24T10:35:50.632243+0000","last_unstale":"2024-03-24T10:35:50.635488+0000","last_undegraded":"2024-03-24T10:35:50.635488+0000","last_fullsized":"2024-03-24T10:35:50.635488+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:44.471627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618657+0000","last_change":"2024-03-24T10:35:50.618657+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618657+0000","last_undegraded":"2024-03-24T10:35:50.618657+0000","last_fullsized":"2024-03-24T10:35:50.618657+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637051+0000","last_change":"2024-03-24T10:35:50.632996+0000","last_active":"2024-03-24T10:35:50.637051+0000","last_peered":"2024-03-24T10:35:50.637051+0000","last_clean":"2024-03-24T10:35:50.637051+0000","last_became_active":"2024-03-24T10:35:50.632669+0000","last_became_peered":"2024-03-24T10:35:50.632669+0000","last_unstale":"2024-03-24T10:35:50.637051+0000","last_undegraded":"2024-03-24T10:35:50.637051+0000","last_fullsized":"2024-03-24T10:35:50.637051+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:45:55.867715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623651+0000","last_change":"2024-03-24T10:35:50.623651+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623651+0000","last_undegraded":"2024-03-24T10:35:50.623651+0000","last_fullsized":"2024-03-24T10:35:50.623651+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.633750+0000","last_change":"2024-03-24T10:35:50.631866+0000","last_active":"2024-03-24T10:35:50.633750+0000","last_peered":"2024-03-24T10:35:50.633750+0000","last_clean":"2024-03-24T10:35:50.633750+0000","last_became_active":"2024-03-24T10:35:50.631578+0000","last_became_peered":"2024-03-24T10:35:50.631578+0000","last_unstale":"2024-03-24T10:35:50.633750+0000","last_undegraded":"2024-03-24T10:35:50.633750+0000","last_fullsized":"2024-03-24T10:35:50.633750+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:57.449362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.628548+0000","last_change":"2024-03-24T10:35:50.628548+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.628548+0000","last_undegraded":"2024-03-24T10:35:50.628548+0000","last_fullsized":"2024-03-24T10:35:50.628548+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632090+0000","last_change":"2024-03-24T10:35:50.629954+0000","last_active":"2024-03-24T10:35:50.632090+0000","last_peered":"2024-03-24T10:35:50.632090+0000","last_clean":"2024-03-24T10:35:50.632090+0000","last_became_active":"2024-03-24T10:35:50.629658+0000","last_became_peered":"2024-03-24T10:35:50.629658+0000","last_unstale":"2024-03-24T10:35:50.632090+0000","last_undegraded":"2024-03-24T10:35:50.632090+0000","last_fullsized":"2024-03-24T10:35:50.632090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:41.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626260+0000","last_change":"2024-03-24T10:35:50.626260+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626260+0000","last_undegraded":"2024-03-24T10:35:50.626260+0000","last_fullsized":"2024-03-24T10:35:50.626260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637838+0000","last_change":"2024-03-24T10:35:50.635109+0000","last_active":"2024-03-24T10:35:50.637838+0000","last_peered":"2024-03-24T10:35:50.637838+0000","last_clean":"2024-03-24T10:35:50.637838+0000","last_became_active":"2024-03-24T10:35:50.634804+0000","last_became_peered":"2024-03-24T10:35:50.634804+0000","last_unstale":"2024-03-24T10:35:50.637838+0000","last_undegraded":"2024-03-24T10:35:50.637838+0000","last_fullsized":"2024-03-24T10:35:50.637838+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:41.493981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618384+0000","last_change":"2024-03-24T10:35:50.618384+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618384+0000","last_undegraded":"2024-03-24T10:35:50.618384+0000","last_fullsized":"2024-03-24T10:35:50.618384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632261+0000","last_change":"2024-03-24T10:35:50.630807+0000","last_active":"2024-03-24T10:35:50.632261+0000","last_peered":"2024-03-24T10:35:50.632261+0000","last_clean":"2024-03-24T10:35:50.632261+0000","last_became_active":"2024-03-24T10:35:50.630512+0000","last_became_peered":"2024-03-24T10:35:50.630512+0000","last_unstale":"2024-03-24T10:35:50.632261+0000","last_undegraded":"2024-03-24T10:35:50.632261+0000","last_fullsized":"2024-03-24T10:35:50.632261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:55.432206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623104+0000","last_change":"2024-03-24T10:35:50.623104+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623104+0000","last_undegraded":"2024-03-24T10:35:50.623104+0000","last_fullsized":"2024-03-24T10:35:50.623104+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639574+0000","last_change":"2024-03-24T10:35:50.638566+0000","last_active":"2024-03-24T10:35:50.639574+0000","last_peered":"2024-03-24T10:35:50.639574+0000","last_clean":"2024-03-24T10:35:50.639574+0000","last_became_active":"2024-03-24T10:35:50.638280+0000","last_became_peered":"2024-03-24T10:35:50.638280+0000","last_unstale":"2024-03-24T10:35:50.639574+0000","last_undegraded":"2024-03-24T10:35:50.639574+0000","last_fullsized":"2024-03-24T10:35:50.639574+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:29.065867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618732+0000","last_change":"2024-03-24T10:35:50.618732+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618732+0000","last_undegraded":"2024-03-24T10:35:50.618732+0000","last_fullsized":"2024-03-24T10:35:50.618732+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634647+0000","last_change":"2024-03-24T10:35:50.631008+0000","last_active":"2024-03-24T10:35:50.634647+0000","last_peered":"2024-03-24T10:35:50.634647+0000","last_clean":"2024-03-24T10:35:50.634647+0000","last_became_active":"2024-03-24T10:35:50.630722+0000","last_became_peered":"2024-03-24T10:35:50.630722+0000","last_unstale":"2024-03-24T10:35:50.634647+0000","last_undegraded":"2024-03-24T10:35:50.634647+0000","last_fullsized":"2024-03-24T10:35:50.634647+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:21.376175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624509+0000","last_change":"2024-03-24T10:35:50.624509+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624509+0000","last_undegraded":"2024-03-24T10:35:50.624509+0000","last_fullsized":"2024-03-24T10:35:50.624509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638327+0000","last_change":"2024-03-24T10:35:50.636330+0000","last_active":"2024-03-24T10:35:50.638327+0000","last_peered":"2024-03-24T10:35:50.638327+0000","last_clean":"2024-03-24T10:35:50.638327+0000","last_became_active":"2024-03-24T10:35:50.636035+0000","last_became_peered":"2024-03-24T10:35:50.636035+0000","last_unstale":"2024-03-24T10:35:50.638327+0000","last_undegraded":"2024-03-24T10:35:50.638327+0000","last_fullsized":"2024-03-24T10:35:50.638327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:54.961171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624517+0000","last_change":"2024-03-24T10:35:50.624517+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624517+0000","last_undegraded":"2024-03-24T10:35:50.624517+0000","last_fullsized":"2024-03-24T10:35:50.624517+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634211+0000","last_change":"2024-03-24T10:35:50.628958+0000","last_active":"2024-03-24T10:35:50.634211+0000","last_peered":"2024-03-24T10:35:50.634211+0000","last_clean":"2024-03-24T10:35:50.634211+0000","last_became_active":"2024-03-24T10:35:50.628661+0000","last_became_peered":"2024-03-24T10:35:50.628661+0000","last_unstale":"2024-03-24T10:35:50.634211+0000","last_undegraded":"2024-03-24T10:35:50.634211+0000","last_fullsized":"2024-03-24T10:35:50.634211+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:01.210812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623387+0000","last_change":"2024-03-24T10:35:50.623387+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623387+0000","last_undegraded":"2024-03-24T10:35:50.623387+0000","last_fullsized":"2024-03-24T10:35:50.623387+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632067+0000","last_change":"2024-03-24T10:35:50.627406+0000","last_active":"2024-03-24T10:35:50.632067+0000","last_peered":"2024-03-24T10:35:50.632067+0000","last_clean":"2024-03-24T10:35:50.632067+0000","last_became_active":"2024-03-24T10:35:50.627116+0000","last_became_peered":"2024-03-24T10:35:50.627116+0000","last_unstale":"2024-03-24T10:35:50.632067+0000","last_undegraded":"2024-03-24T10:35:50.632067+0000","last_fullsized":"2024-03-24T10:35:50.632067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:44.632873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630039+0000","last_change":"2024-03-24T10:35:50.630039+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630039+0000","last_undegraded":"2024-03-24T10:35:50.630039+0000","last_fullsized":"2024-03-24T10:35:50.630039+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638602+0000","last_change":"2024-03-24T10:35:50.635584+0000","last_active":"2024-03-24T10:35:50.638602+0000","last_peered":"2024-03-24T10:35:50.638602+0000","last_clean":"2024-03-24T10:35:50.638602+0000","last_became_active":"2024-03-24T10:35:50.635280+0000","last_became_peered":"2024-03-24T10:35:50.635280+0000","last_unstale":"2024-03-24T10:35:50.638602+0000","last_undegraded":"2024-03-24T10:35:50.638602+0000","last_fullsized":"2024-03-24T10:35:50.638602+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:27:14.345670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622626+0000","last_change":"2024-03-24T10:35:50.622626+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622626+0000","last_undegraded":"2024-03-24T10:35:50.622626+0000","last_fullsized":"2024-03-24T10:35:50.622626+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635400+0000","last_change":"2024-03-24T10:35:50.631769+0000","last_active":"2024-03-24T10:35:50.635400+0000","last_peered":"2024-03-24T10:35:50.635400+0000","last_clean":"2024-03-24T10:35:50.635400+0000","last_became_active":"2024-03-24T10:35:50.631453+0000","last_became_peered":"2024-03-24T10:35:50.631453+0000","last_unstale":"2024-03-24T10:35:50.635400+0000","last_undegraded":"2024-03-24T10:35:50.635400+0000","last_fullsized":"2024-03-24T10:35:50.635400+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:11.009937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626484+0000","last_change":"2024-03-24T10:35:50.626484+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626484+0000","last_undegraded":"2024-03-24T10:35:50.626484+0000","last_fullsized":"2024-03-24T10:35:50.626484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638457+0000","last_change":"2024-03-24T10:35:50.636812+0000","last_active":"2024-03-24T10:35:50.638457+0000","last_peered":"2024-03-24T10:35:50.638457+0000","last_clean":"2024-03-24T10:35:50.638457+0000","last_became_active":"2024-03-24T10:35:50.636520+0000","last_became_peered":"2024-03-24T10:35:50.636520+0000","last_unstale":"2024-03-24T10:35:50.638457+0000","last_undegraded":"2024-03-24T10:35:50.638457+0000","last_fullsized":"2024-03-24T10:35:50.638457+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:16.151719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.622920+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:50.622920+0000","last_peered":"2024-03-24T10:35:50.622920+0000","last_clean":"2024-03-24T10:35:50.622920+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:50.622920+0000","last_undegraded":"2024-03-24T10:35:50.622920+0000","last_fullsized":"2024-03-24T10:35:50.622920+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":227616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:35:55.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:35:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.185+0000 7f35283f4640 1 Processor -- start 2024-03-24T10:35:55.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 -- start start 2024-03-24T10:35:55.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35200d2bb0 0x7f35200b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f35200b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352015e770 con 0x7f352015e390 2024-03-24T10:35:55.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f35200b3190 con 0x7f35200d2bb0 2024-03-24T10:35:55.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.186+0000 7f35283f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f35200d4e40 con 0x7f35200d1000 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35200d2bb0 0x7f35200b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35200d2bb0 0x7f35200b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38612/0 (socket says 172.21.15.1:38612) 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 -- 172.21.15.1:0/3949295645 learned_addr learned my addr 172.21.15.1:0/3949295645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f3526991640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f35200b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 -- 172.21.15.1:0/3949295645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 msgr2=0x7f35200b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f35200b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 -- 172.21.15.1:0/3949295645 >> v1:172.21.15.3:6789/0 conn(0x7f352015e390 legacy=0x7f35200d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f352598f640 1 -- 172.21.15.1:0/3949295645 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35200d1e40 con 0x7f35200d2bb0 2024-03-24T10:35:55.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.187+0000 7f3526991640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f35200b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:55.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f352598f640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f35200b3b90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3510009870 tx=0x7f351002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15167e649b2e4dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f352518e640 1 -- 172.21.15.1:0/3949295645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35100317c0 con 0x7f35200d2bb0 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f352518e640 1 -- 172.21.15.1:0/3949295645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3510031920 con 0x7f35200d2bb0 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f352518e640 1 -- 172.21.15.1:0/3949295645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510031cb0 con 0x7f35200d2bb0 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 -- 172.21.15.1:0/3949295645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 msgr2=0x7f35200b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f35200b3b90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3510009870 tx=0x7f351002b860 comp rx=0 tx=0).stop 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 -- 172.21.15.1:0/3949295645 shutdown_connections 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f35200b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 --2- 172.21.15.1:0/3949295645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f35200b3b90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.188+0000 7f35283f4640 1 -- 172.21.15.1:0/3949295645 >> 172.21.15.1:0/3949295645 conn(0x7f35200d3ec0 msgr2=0x7f35200d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.189+0000 7f35283f4640 1 -- 172.21.15.1:0/3949295645 shutdown_connections 2024-03-24T10:35:55.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.189+0000 7f35283f4640 1 -- 172.21.15.1:0/3949295645 wait complete. 2024-03-24T10:35:55.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 Processor -- start 2024-03-24T10:35:55.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 -- start start 2024-03-24T10:35:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f3520152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 0x7f35200b9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f352009e800 con 0x7f35200d2bb0 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f352009dc10 con 0x7f352015e390 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f35283f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3520152880 con 0x7f35200d1000 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f352598f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f352598f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38620/0 (socket says 172.21.15.1:38620) 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.190+0000 7f352598f640 1 -- 172.21.15.1:0/2087371344 learned_addr learned my addr 172.21.15.1:0/2087371344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3526190640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f3520152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3526991640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 0x7f35200b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 -- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 msgr2=0x7f3520152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f3520152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 -- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 msgr2=0x7f35200b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 0x7f35200b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 -- 172.21.15.1:0/2087371344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35100094d0 con 0x7f35200d2bb0 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3526991640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 0x7f35200b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f352598f640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3510000c00 tx=0x7f35100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510032860 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3510032a00 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352009de60 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510032db0 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.191+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352009e420 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.192+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f352004f4e0 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.193+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f35100453f0 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.193+0000 7f3516ffd640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 0x7f350406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.193+0000 7f3526190640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 0x7f350406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.193+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f35100ac460 con 0x7f35200d2bb0 2024-03-24T10:35:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.194+0000 7f3526190640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 0x7f350406dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f351c007fd0 tx=0x7f351c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.197+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3510076f40 con 0x7f35200d2bb0 2024-03-24T10:35:55.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.440+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35200b3320 con 0x7f350406b810 2024-03-24T10:35:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.446+0000 7f3516ffd640 1 -- 172.21.15.1:0/2087371344 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246641 (secure 0 0 0) 0x7f351c0050d0 con 0x7f350406b810 2024-03-24T10:35:55.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:55.450 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 msgr2=0x7f350406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 0x7f350406dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f351c007fd0 tx=0x7f351c009040 comp rx=0 tx=0).stop 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 msgr2=0x7f352009ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3510000c00 tx=0x7f35100046c0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 shutdown_connections 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f350406b810 0x7f350406dcd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f352015e390 0x7f35200b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35200d2bb0 0x7f352009ec20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 --2- 172.21.15.1:0/2087371344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f35200d1000 0x7f3520152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.449+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 >> 172.21.15.1:0/2087371344 conn(0x7f35200d3ec0 msgr2=0x7f352006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.450+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 shutdown_connections 2024-03-24T10:35:55.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.450+0000 7f35283f4640 1 -- 172.21.15.1:0/2087371344 wait complete. 2024-03-24T10:35:55.464 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-03-24T10:35:54.539396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238424,"kb_used_data":5596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736296,"statfs":{"total":773094113280,"available":772849967104,"internally_reserved":0,"allocated":5730304,"data_stored":3045589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931065"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619572+0000","last_change":"2024-03-24T10:35:50.619572+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619572+0000","last_undegraded":"2024-03-24T10:35:50.619572+0000","last_fullsized":"2024-03-24T10:35:50.619572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622464+0000","last_change":"2024-03-24T10:35:50.622464+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622464+0000","last_undegraded":"2024-03-24T10:35:50.622464+0000","last_fullsized":"2024-03-24T10:35:50.622464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627689+0000","last_change":"2024-03-24T10:35:50.627689+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627689+0000","last_undegraded":"2024-03-24T10:35:50.627689+0000","last_fullsized":"2024-03-24T10:35:50.627689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621316+0000","last_change":"2024-03-24T10:35:50.621316+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621316+0000","last_undegraded":"2024-03-24T10:35:50.621316+0000","last_fullsized":"2024-03-24T10:35:50.621316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624565+0000","last_change":"2024-03-24T10:35:50.624565+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624565+0000","last_undegraded":"2024-03-24T10:35:50.624565+0000","last_fullsized":"2024-03-24T10:35:50.624565+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619946+0000","last_change":"2024-03-24T10:35:50.619946+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619946+0000","last_undegraded":"2024-03-24T10:35:50.619946+0000","last_fullsized":"2024-03-24T10:35:50.619946+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624479+0000","last_change":"2024-03-24T10:35:50.624479+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624479+0000","last_undegraded":"2024-03-24T10:35:50.624479+0000","last_fullsized":"2024-03-24T10:35:50.624479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624154+0000","last_change":"2024-03-24T10:35:50.624154+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624154+0000","last_undegraded":"2024-03-24T10:35:50.624154+0000","last_fullsized":"2024-03-24T10:35:50.624154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623889+0000","last_change":"2024-03-24T10:35:50.623889+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623889+0000","last_undegraded":"2024-03-24T10:35:50.623889+0000","last_fullsized":"2024-03-24T10:35:50.623889+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619433+0000","last_change":"2024-03-24T10:35:50.619433+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619433+0000","last_undegraded":"2024-03-24T10:35:50.619433+0000","last_fullsized":"2024-03-24T10:35:50.619433+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619034+0000","last_change":"2024-03-24T10:35:50.619034+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619034+0000","last_undegraded":"2024-03-24T10:35:50.619034+0000","last_fullsized":"2024-03-24T10:35:50.619034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622027+0000","last_change":"2024-03-24T10:35:50.622027+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622027+0000","last_undegraded":"2024-03-24T10:35:50.622027+0000","last_fullsized":"2024-03-24T10:35:50.622027+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619363+0000","last_change":"2024-03-24T10:35:50.619363+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619363+0000","last_undegraded":"2024-03-24T10:35:50.619363+0000","last_fullsized":"2024-03-24T10:35:50.619363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624300+0000","last_change":"2024-03-24T10:35:50.624300+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624300+0000","last_undegraded":"2024-03-24T10:35:50.624300+0000","last_fullsized":"2024-03-24T10:35:50.624300+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621221+0000","last_change":"2024-03-24T10:35:50.621221+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621221+0000","last_undegraded":"2024-03-24T10:35:50.621221+0000","last_fullsized":"2024-03-24T10:35:50.621221+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618425+0000","last_change":"2024-03-24T10:35:50.618425+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618425+0000","last_undegraded":"2024-03-24T10:35:50.618425+0000","last_fullsized":"2024-03-24T10:35:50.618425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627699+0000","last_change":"2024-03-24T10:35:50.627699+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627699+0000","last_undegraded":"2024-03-24T10:35:50.627699+0000","last_fullsized":"2024-03-24T10:35:50.627699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622054+0000","last_change":"2024-03-24T10:35:50.622054+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622054+0000","last_undegraded":"2024-03-24T10:35:50.622054+0000","last_fullsized":"2024-03-24T10:35:50.622054+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627222+0000","last_change":"2024-03-24T10:35:50.627222+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627222+0000","last_undegraded":"2024-03-24T10:35:50.627222+0000","last_fullsized":"2024-03-24T10:35:50.627222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627419+0000","last_change":"2024-03-24T10:35:50.627419+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627419+0000","last_undegraded":"2024-03-24T10:35:50.627419+0000","last_fullsized":"2024-03-24T10:35:50.627419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625486+0000","last_change":"2024-03-24T10:35:50.625486+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625486+0000","last_undegraded":"2024-03-24T10:35:50.625486+0000","last_fullsized":"2024-03-24T10:35:50.625486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622677+0000","last_change":"2024-03-24T10:35:50.622677+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622677+0000","last_undegraded":"2024-03-24T10:35:50.622677+0000","last_fullsized":"2024-03-24T10:35:50.622677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618841+0000","last_change":"2024-03-24T10:35:50.618841+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618841+0000","last_undegraded":"2024-03-24T10:35:50.618841+0000","last_fullsized":"2024-03-24T10:35:50.618841+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622708+0000","last_change":"2024-03-24T10:35:50.622708+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622708+0000","last_undegraded":"2024-03-24T10:35:50.622708+0000","last_fullsized":"2024-03-24T10:35:50.622708+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625709+0000","last_change":"2024-03-24T10:35:50.625709+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625709+0000","last_undegraded":"2024-03-24T10:35:50.625709+0000","last_fullsized":"2024-03-24T10:35:50.625709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625691+0000","last_change":"2024-03-24T10:35:50.625691+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625691+0000","last_undegraded":"2024-03-24T10:35:50.625691+0000","last_fullsized":"2024-03-24T10:35:50.625691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618448+0000","last_change":"2024-03-24T10:35:50.618448+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618448+0000","last_undegraded":"2024-03-24T10:35:50.618448+0000","last_fullsized":"2024-03-24T10:35:50.618448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626365+0000","last_change":"2024-03-24T10:35:50.626365+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626365+0000","last_undegraded":"2024-03-24T10:35:50.626365+0000","last_fullsized":"2024-03-24T10:35:50.626365+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630750+0000","last_change":"2024-03-24T10:35:50.630750+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630750+0000","last_undegraded":"2024-03-24T10:35:50.630750+0000","last_fullsized":"2024-03-24T10:35:50.630750+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621203+0000","last_change":"2024-03-24T10:35:50.621203+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621203+0000","last_undegraded":"2024-03-24T10:35:50.621203+0000","last_fullsized":"2024-03-24T10:35:50.621203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623326+0000","last_change":"2024-03-24T10:35:50.623326+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623326+0000","last_undegraded":"2024-03-24T10:35:50.623326+0000","last_fullsized":"2024-03-24T10:35:50.623326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624597+0000","last_change":"2024-03-24T10:35:50.624597+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624597+0000","last_undegraded":"2024-03-24T10:35:50.624597+0000","last_fullsized":"2024-03-24T10:35:50.624597+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621815+0000","last_change":"2024-03-24T10:35:50.621815+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621815+0000","last_undegraded":"2024-03-24T10:35:50.621815+0000","last_fullsized":"2024-03-24T10:35:50.621815+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638164+0000","last_change":"2024-03-24T10:35:50.635274+0000","last_active":"2024-03-24T10:35:50.638164+0000","last_peered":"2024-03-24T10:35:50.638164+0000","last_clean":"2024-03-24T10:35:50.638164+0000","last_became_active":"2024-03-24T10:35:50.634934+0000","last_became_peered":"2024-03-24T10:35:50.634934+0000","last_unstale":"2024-03-24T10:35:50.638164+0000","last_undegraded":"2024-03-24T10:35:50.638164+0000","last_fullsized":"2024-03-24T10:35:50.638164+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:19.768286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622233+0000","last_change":"2024-03-24T10:35:50.622233+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622233+0000","last_undegraded":"2024-03-24T10:35:50.622233+0000","last_fullsized":"2024-03-24T10:35:50.622233+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634816+0000","last_change":"2024-03-24T10:35:50.633891+0000","last_active":"2024-03-24T10:35:50.634816+0000","last_peered":"2024-03-24T10:35:50.634816+0000","last_clean":"2024-03-24T10:35:50.634816+0000","last_became_active":"2024-03-24T10:35:50.633604+0000","last_became_peered":"2024-03-24T10:35:50.633604+0000","last_unstale":"2024-03-24T10:35:50.634816+0000","last_undegraded":"2024-03-24T10:35:50.634816+0000","last_fullsized":"2024-03-24T10:35:50.634816+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:45.059275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626163+0000","last_change":"2024-03-24T10:35:50.626163+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626163+0000","last_undegraded":"2024-03-24T10:35:50.626163+0000","last_fullsized":"2024-03-24T10:35:50.626163+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638820+0000","last_change":"2024-03-24T10:35:50.637492+0000","last_active":"2024-03-24T10:35:50.638820+0000","last_peered":"2024-03-24T10:35:50.638820+0000","last_clean":"2024-03-24T10:35:50.638820+0000","last_became_active":"2024-03-24T10:35:50.637225+0000","last_became_peered":"2024-03-24T10:35:50.637225+0000","last_unstale":"2024-03-24T10:35:50.638820+0000","last_undegraded":"2024-03-24T10:35:50.638820+0000","last_fullsized":"2024-03-24T10:35:50.638820+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:12.007000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625329+0000","last_change":"2024-03-24T10:35:50.625329+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625329+0000","last_undegraded":"2024-03-24T10:35:50.625329+0000","last_fullsized":"2024-03-24T10:35:50.625329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638326+0000","last_change":"2024-03-24T10:35:50.632990+0000","last_active":"2024-03-24T10:35:50.638326+0000","last_peered":"2024-03-24T10:35:50.638326+0000","last_clean":"2024-03-24T10:35:50.638326+0000","last_became_active":"2024-03-24T10:35:50.632635+0000","last_became_peered":"2024-03-24T10:35:50.632635+0000","last_unstale":"2024-03-24T10:35:50.638326+0000","last_undegraded":"2024-03-24T10:35:50.638326+0000","last_fullsized":"2024-03-24T10:35:50.638326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:47.712468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623984+0000","last_change":"2024-03-24T10:35:50.623984+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623984+0000","last_undegraded":"2024-03-24T10:35:50.623984+0000","last_fullsized":"2024-03-24T10:35:50.623984+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639222+0000","last_change":"2024-03-24T10:35:50.638378+0000","last_active":"2024-03-24T10:35:50.639222+0000","last_peered":"2024-03-24T10:35:50.639222+0000","last_clean":"2024-03-24T10:35:50.639222+0000","last_became_active":"2024-03-24T10:35:50.638077+0000","last_became_peered":"2024-03-24T10:35:50.638077+0000","last_unstale":"2024-03-24T10:35:50.639222+0000","last_undegraded":"2024-03-24T10:35:50.639222+0000","last_fullsized":"2024-03-24T10:35:50.639222+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:48:48.966513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621207+0000","last_change":"2024-03-24T10:35:50.621207+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621207+0000","last_undegraded":"2024-03-24T10:35:50.621207+0000","last_fullsized":"2024-03-24T10:35:50.621207+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638629+0000","last_change":"2024-03-24T10:35:50.636470+0000","last_active":"2024-03-24T10:35:50.638629+0000","last_peered":"2024-03-24T10:35:50.638629+0000","last_clean":"2024-03-24T10:35:50.638629+0000","last_became_active":"2024-03-24T10:35:50.636139+0000","last_became_peered":"2024-03-24T10:35:50.636139+0000","last_unstale":"2024-03-24T10:35:50.638629+0000","last_undegraded":"2024-03-24T10:35:50.638629+0000","last_fullsized":"2024-03-24T10:35:50.638629+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:55.032277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624799+0000","last_change":"2024-03-24T10:35:50.624799+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624799+0000","last_undegraded":"2024-03-24T10:35:50.624799+0000","last_fullsized":"2024-03-24T10:35:50.624799+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637837+0000","last_change":"2024-03-24T10:35:50.635522+0000","last_active":"2024-03-24T10:35:50.637837+0000","last_peered":"2024-03-24T10:35:50.637837+0000","last_clean":"2024-03-24T10:35:50.637837+0000","last_became_active":"2024-03-24T10:35:50.635036+0000","last_became_peered":"2024-03-24T10:35:50.635036+0000","last_unstale":"2024-03-24T10:35:50.637837+0000","last_undegraded":"2024-03-24T10:35:50.637837+0000","last_fullsized":"2024-03-24T10:35:50.637837+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:55.117568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621406+0000","last_change":"2024-03-24T10:35:50.621406+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621406+0000","last_undegraded":"2024-03-24T10:35:50.621406+0000","last_fullsized":"2024-03-24T10:35:50.621406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638491+0000","last_change":"2024-03-24T10:35:50.636110+0000","last_active":"2024-03-24T10:35:50.638491+0000","last_peered":"2024-03-24T10:35:50.638491+0000","last_clean":"2024-03-24T10:35:50.638491+0000","last_became_active":"2024-03-24T10:35:50.635804+0000","last_became_peered":"2024-03-24T10:35:50.635804+0000","last_unstale":"2024-03-24T10:35:50.638491+0000","last_undegraded":"2024-03-24T10:35:50.638491+0000","last_fullsized":"2024-03-24T10:35:50.638491+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:26.730054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623498+0000","last_change":"2024-03-24T10:35:50.623498+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623498+0000","last_undegraded":"2024-03-24T10:35:50.623498+0000","last_fullsized":"2024-03-24T10:35:50.623498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634675+0000","last_change":"2024-03-24T10:35:50.630099+0000","last_active":"2024-03-24T10:35:50.634675+0000","last_peered":"2024-03-24T10:35:50.634675+0000","last_clean":"2024-03-24T10:35:50.634675+0000","last_became_active":"2024-03-24T10:35:50.629665+0000","last_became_peered":"2024-03-24T10:35:50.629665+0000","last_unstale":"2024-03-24T10:35:50.634675+0000","last_undegraded":"2024-03-24T10:35:50.634675+0000","last_fullsized":"2024-03-24T10:35:50.634675+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:10.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622768+0000","last_change":"2024-03-24T10:35:50.622768+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622768+0000","last_undegraded":"2024-03-24T10:35:50.622768+0000","last_fullsized":"2024-03-24T10:35:50.622768+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639380+0000","last_change":"2024-03-24T10:35:50.638305+0000","last_active":"2024-03-24T10:35:50.639380+0000","last_peered":"2024-03-24T10:35:50.639380+0000","last_clean":"2024-03-24T10:35:50.639380+0000","last_became_active":"2024-03-24T10:35:50.637768+0000","last_became_peered":"2024-03-24T10:35:50.637768+0000","last_unstale":"2024-03-24T10:35:50.639380+0000","last_undegraded":"2024-03-24T10:35:50.639380+0000","last_fullsized":"2024-03-24T10:35:50.639380+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:33:50.611475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623833+0000","last_change":"2024-03-24T10:35:50.623833+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623833+0000","last_undegraded":"2024-03-24T10:35:50.623833+0000","last_fullsized":"2024-03-24T10:35:50.623833+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634930+0000","last_change":"2024-03-24T10:35:50.631873+0000","last_active":"2024-03-24T10:35:50.634930+0000","last_peered":"2024-03-24T10:35:50.634930+0000","last_clean":"2024-03-24T10:35:50.634930+0000","last_became_active":"2024-03-24T10:35:50.631567+0000","last_became_peered":"2024-03-24T10:35:50.631567+0000","last_unstale":"2024-03-24T10:35:50.634930+0000","last_undegraded":"2024-03-24T10:35:50.634930+0000","last_fullsized":"2024-03-24T10:35:50.634930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:29.632034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623884+0000","last_change":"2024-03-24T10:35:50.623884+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623884+0000","last_undegraded":"2024-03-24T10:35:50.623884+0000","last_fullsized":"2024-03-24T10:35:50.623884+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632135+0000","last_change":"2024-03-24T10:35:50.626864+0000","last_active":"2024-03-24T10:35:50.632135+0000","last_peered":"2024-03-24T10:35:50.632135+0000","last_clean":"2024-03-24T10:35:50.632135+0000","last_became_active":"2024-03-24T10:35:50.626556+0000","last_became_peered":"2024-03-24T10:35:50.626556+0000","last_unstale":"2024-03-24T10:35:50.632135+0000","last_undegraded":"2024-03-24T10:35:50.632135+0000","last_fullsized":"2024-03-24T10:35:50.632135+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:40.363003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624072+0000","last_change":"2024-03-24T10:35:50.624072+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624072+0000","last_undegraded":"2024-03-24T10:35:50.624072+0000","last_fullsized":"2024-03-24T10:35:50.624072+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638533+0000","last_change":"2024-03-24T10:35:50.637003+0000","last_active":"2024-03-24T10:35:50.638533+0000","last_peered":"2024-03-24T10:35:50.638533+0000","last_clean":"2024-03-24T10:35:50.638533+0000","last_became_active":"2024-03-24T10:35:50.636716+0000","last_became_peered":"2024-03-24T10:35:50.636716+0000","last_unstale":"2024-03-24T10:35:50.638533+0000","last_undegraded":"2024-03-24T10:35:50.638533+0000","last_fullsized":"2024-03-24T10:35:50.638533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:13.161270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625445+0000","last_change":"2024-03-24T10:35:50.625445+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625445+0000","last_undegraded":"2024-03-24T10:35:50.625445+0000","last_fullsized":"2024-03-24T10:35:50.625445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638748+0000","last_change":"2024-03-24T10:35:50.637095+0000","last_active":"2024-03-24T10:35:50.638748+0000","last_peered":"2024-03-24T10:35:50.638748+0000","last_clean":"2024-03-24T10:35:50.638748+0000","last_became_active":"2024-03-24T10:35:50.636717+0000","last_became_peered":"2024-03-24T10:35:50.636717+0000","last_unstale":"2024-03-24T10:35:50.638748+0000","last_undegraded":"2024-03-24T10:35:50.638748+0000","last_fullsized":"2024-03-24T10:35:50.638748+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.035067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625235+0000","last_change":"2024-03-24T10:35:50.625235+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625235+0000","last_undegraded":"2024-03-24T10:35:50.625235+0000","last_fullsized":"2024-03-24T10:35:50.625235+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638235+0000","last_change":"2024-03-24T10:35:50.637013+0000","last_active":"2024-03-24T10:35:50.638235+0000","last_peered":"2024-03-24T10:35:50.638235+0000","last_clean":"2024-03-24T10:35:50.638235+0000","last_became_active":"2024-03-24T10:35:50.636281+0000","last_became_peered":"2024-03-24T10:35:50.636281+0000","last_unstale":"2024-03-24T10:35:50.638235+0000","last_undegraded":"2024-03-24T10:35:50.638235+0000","last_fullsized":"2024-03-24T10:35:50.638235+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:40.428655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.629902+0000","last_change":"2024-03-24T10:35:50.629902+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.629902+0000","last_undegraded":"2024-03-24T10:35:50.629902+0000","last_fullsized":"2024-03-24T10:35:50.629902+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.630082+0000","last_change":"2024-03-24T10:35:50.626837+0000","last_active":"2024-03-24T10:35:50.630082+0000","last_peered":"2024-03-24T10:35:50.630082+0000","last_clean":"2024-03-24T10:35:50.630082+0000","last_became_active":"2024-03-24T10:35:50.626538+0000","last_became_peered":"2024-03-24T10:35:50.626538+0000","last_unstale":"2024-03-24T10:35:50.630082+0000","last_undegraded":"2024-03-24T10:35:50.630082+0000","last_fullsized":"2024-03-24T10:35:50.630082+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:55.782184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619769+0000","last_change":"2024-03-24T10:35:50.619769+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619769+0000","last_undegraded":"2024-03-24T10:35:50.619769+0000","last_fullsized":"2024-03-24T10:35:50.619769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639683+0000","last_change":"2024-03-24T10:35:50.638452+0000","last_active":"2024-03-24T10:35:50.639683+0000","last_peered":"2024-03-24T10:35:50.639683+0000","last_clean":"2024-03-24T10:35:50.639683+0000","last_became_active":"2024-03-24T10:35:50.636353+0000","last_became_peered":"2024-03-24T10:35:50.636353+0000","last_unstale":"2024-03-24T10:35:50.639683+0000","last_undegraded":"2024-03-24T10:35:50.639683+0000","last_fullsized":"2024-03-24T10:35:50.639683+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:20.996736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618403+0000","last_change":"2024-03-24T10:35:50.618403+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618403+0000","last_undegraded":"2024-03-24T10:35:50.618403+0000","last_fullsized":"2024-03-24T10:35:50.618403+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.629210+0000","last_change":"2024-03-24T10:35:50.628002+0000","last_active":"2024-03-24T10:35:50.629210+0000","last_peered":"2024-03-24T10:35:50.629210+0000","last_clean":"2024-03-24T10:35:50.629210+0000","last_became_active":"2024-03-24T10:35:50.627615+0000","last_became_peered":"2024-03-24T10:35:50.627615+0000","last_unstale":"2024-03-24T10:35:50.629210+0000","last_undegraded":"2024-03-24T10:35:50.629210+0000","last_fullsized":"2024-03-24T10:35:50.629210+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:04:01.756032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624501+0000","last_change":"2024-03-24T10:35:50.624501+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624501+0000","last_undegraded":"2024-03-24T10:35:50.624501+0000","last_fullsized":"2024-03-24T10:35:50.624501+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635488+0000","last_change":"2024-03-24T10:35:50.632552+0000","last_active":"2024-03-24T10:35:50.635488+0000","last_peered":"2024-03-24T10:35:50.635488+0000","last_clean":"2024-03-24T10:35:50.635488+0000","last_became_active":"2024-03-24T10:35:50.632243+0000","last_became_peered":"2024-03-24T10:35:50.632243+0000","last_unstale":"2024-03-24T10:35:50.635488+0000","last_undegraded":"2024-03-24T10:35:50.635488+0000","last_fullsized":"2024-03-24T10:35:50.635488+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:44.471627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618657+0000","last_change":"2024-03-24T10:35:50.618657+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618657+0000","last_undegraded":"2024-03-24T10:35:50.618657+0000","last_fullsized":"2024-03-24T10:35:50.618657+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637051+0000","last_change":"2024-03-24T10:35:50.632996+0000","last_active":"2024-03-24T10:35:50.637051+0000","last_peered":"2024-03-24T10:35:50.637051+0000","last_clean":"2024-03-24T10:35:50.637051+0000","last_became_active":"2024-03-24T10:35:50.632669+0000","last_became_peered":"2024-03-24T10:35:50.632669+0000","last_unstale":"2024-03-24T10:35:50.637051+0000","last_undegraded":"2024-03-24T10:35:50.637051+0000","last_fullsized":"2024-03-24T10:35:50.637051+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:45:55.867715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623651+0000","last_change":"2024-03-24T10:35:50.623651+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623651+0000","last_undegraded":"2024-03-24T10:35:50.623651+0000","last_fullsized":"2024-03-24T10:35:50.623651+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.633750+0000","last_change":"2024-03-24T10:35:50.631866+0000","last_active":"2024-03-24T10:35:50.633750+0000","last_peered":"2024-03-24T10:35:50.633750+0000","last_clean":"2024-03-24T10:35:50.633750+0000","last_became_active":"2024-03-24T10:35:50.631578+0000","last_became_peered":"2024-03-24T10:35:50.631578+0000","last_unstale":"2024-03-24T10:35:50.633750+0000","last_undegraded":"2024-03-24T10:35:50.633750+0000","last_fullsized":"2024-03-24T10:35:50.633750+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:57.449362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.628548+0000","last_change":"2024-03-24T10:35:50.628548+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.628548+0000","last_undegraded":"2024-03-24T10:35:50.628548+0000","last_fullsized":"2024-03-24T10:35:50.628548+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632090+0000","last_change":"2024-03-24T10:35:50.629954+0000","last_active":"2024-03-24T10:35:50.632090+0000","last_peered":"2024-03-24T10:35:50.632090+0000","last_clean":"2024-03-24T10:35:50.632090+0000","last_became_active":"2024-03-24T10:35:50.629658+0000","last_became_peered":"2024-03-24T10:35:50.629658+0000","last_unstale":"2024-03-24T10:35:50.632090+0000","last_undegraded":"2024-03-24T10:35:50.632090+0000","last_fullsized":"2024-03-24T10:35:50.632090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:41.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626260+0000","last_change":"2024-03-24T10:35:50.626260+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626260+0000","last_undegraded":"2024-03-24T10:35:50.626260+0000","last_fullsized":"2024-03-24T10:35:50.626260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637838+0000","last_change":"2024-03-24T10:35:50.635109+0000","last_active":"2024-03-24T10:35:50.637838+0000","last_peered":"2024-03-24T10:35:50.637838+0000","last_clean":"2024-03-24T10:35:50.637838+0000","last_became_active":"2024-03-24T10:35:50.634804+0000","last_became_peered":"2024-03-24T10:35:50.634804+0000","last_unstale":"2024-03-24T10:35:50.637838+0000","last_undegraded":"2024-03-24T10:35:50.637838+0000","last_fullsized":"2024-03-24T10:35:50.637838+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:41.493981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618384+0000","last_change":"2024-03-24T10:35:50.618384+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618384+0000","last_undegraded":"2024-03-24T10:35:50.618384+0000","last_fullsized":"2024-03-24T10:35:50.618384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632261+0000","last_change":"2024-03-24T10:35:50.630807+0000","last_active":"2024-03-24T10:35:50.632261+0000","last_peered":"2024-03-24T10:35:50.632261+0000","last_clean":"2024-03-24T10:35:50.632261+0000","last_became_active":"2024-03-24T10:35:50.630512+0000","last_became_peered":"2024-03-24T10:35:50.630512+0000","last_unstale":"2024-03-24T10:35:50.632261+0000","last_undegraded":"2024-03-24T10:35:50.632261+0000","last_fullsized":"2024-03-24T10:35:50.632261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:55.432206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623104+0000","last_change":"2024-03-24T10:35:50.623104+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623104+0000","last_undegraded":"2024-03-24T10:35:50.623104+0000","last_fullsized":"2024-03-24T10:35:50.623104+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639574+0000","last_change":"2024-03-24T10:35:50.638566+0000","last_active":"2024-03-24T10:35:50.639574+0000","last_peered":"2024-03-24T10:35:50.639574+0000","last_clean":"2024-03-24T10:35:50.639574+0000","last_became_active":"2024-03-24T10:35:50.638280+0000","last_became_peered":"2024-03-24T10:35:50.638280+0000","last_unstale":"2024-03-24T10:35:50.639574+0000","last_undegraded":"2024-03-24T10:35:50.639574+0000","last_fullsized":"2024-03-24T10:35:50.639574+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:29.065867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618732+0000","last_change":"2024-03-24T10:35:50.618732+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618732+0000","last_undegraded":"2024-03-24T10:35:50.618732+0000","last_fullsized":"2024-03-24T10:35:50.618732+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634647+0000","last_change":"2024-03-24T10:35:50.631008+0000","last_active":"2024-03-24T10:35:50.634647+0000","last_peered":"2024-03-24T10:35:50.634647+0000","last_clean":"2024-03-24T10:35:50.634647+0000","last_became_active":"2024-03-24T10:35:50.630722+0000","last_became_peered":"2024-03-24T10:35:50.630722+0000","last_unstale":"2024-03-24T10:35:50.634647+0000","last_undegraded":"2024-03-24T10:35:50.634647+0000","last_fullsized":"2024-03-24T10:35:50.634647+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:21.376175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624509+0000","last_change":"2024-03-24T10:35:50.624509+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624509+0000","last_undegraded":"2024-03-24T10:35:50.624509+0000","last_fullsized":"2024-03-24T10:35:50.624509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638327+0000","last_change":"2024-03-24T10:35:50.636330+0000","last_active":"2024-03-24T10:35:50.638327+0000","last_peered":"2024-03-24T10:35:50.638327+0000","last_clean":"2024-03-24T10:35:50.638327+0000","last_became_active":"2024-03-24T10:35:50.636035+0000","last_became_peered":"2024-03-24T10:35:50.636035+0000","last_unstale":"2024-03-24T10:35:50.638327+0000","last_undegraded":"2024-03-24T10:35:50.638327+0000","last_fullsized":"2024-03-24T10:35:50.638327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:54.961171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624517+0000","last_change":"2024-03-24T10:35:50.624517+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624517+0000","last_undegraded":"2024-03-24T10:35:50.624517+0000","last_fullsized":"2024-03-24T10:35:50.624517+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634211+0000","last_change":"2024-03-24T10:35:50.628958+0000","last_active":"2024-03-24T10:35:50.634211+0000","last_peered":"2024-03-24T10:35:50.634211+0000","last_clean":"2024-03-24T10:35:50.634211+0000","last_became_active":"2024-03-24T10:35:50.628661+0000","last_became_peered":"2024-03-24T10:35:50.628661+0000","last_unstale":"2024-03-24T10:35:50.634211+0000","last_undegraded":"2024-03-24T10:35:50.634211+0000","last_fullsized":"2024-03-24T10:35:50.634211+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:01.210812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623387+0000","last_change":"2024-03-24T10:35:50.623387+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623387+0000","last_undegraded":"2024-03-24T10:35:50.623387+0000","last_fullsized":"2024-03-24T10:35:50.623387+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632067+0000","last_change":"2024-03-24T10:35:50.627406+0000","last_active":"2024-03-24T10:35:50.632067+0000","last_peered":"2024-03-24T10:35:50.632067+0000","last_clean":"2024-03-24T10:35:50.632067+0000","last_became_active":"2024-03-24T10:35:50.627116+0000","last_became_peered":"2024-03-24T10:35:50.627116+0000","last_unstale":"2024-03-24T10:35:50.632067+0000","last_undegraded":"2024-03-24T10:35:50.632067+0000","last_fullsized":"2024-03-24T10:35:50.632067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:44.632873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630039+0000","last_change":"2024-03-24T10:35:50.630039+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630039+0000","last_undegraded":"2024-03-24T10:35:50.630039+0000","last_fullsized":"2024-03-24T10:35:50.630039+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638602+0000","last_change":"2024-03-24T10:35:50.635584+0000","last_active":"2024-03-24T10:35:50.638602+0000","last_peered":"2024-03-24T10:35:50.638602+0000","last_clean":"2024-03-24T10:35:50.638602+0000","last_became_active":"2024-03-24T10:35:50.635280+0000","last_became_peered":"2024-03-24T10:35:50.635280+0000","last_unstale":"2024-03-24T10:35:50.638602+0000","last_undegraded":"2024-03-24T10:35:50.638602+0000","last_fullsized":"2024-03-24T10:35:50.638602+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:27:14.345670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622626+0000","last_change":"2024-03-24T10:35:50.622626+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622626+0000","last_undegraded":"2024-03-24T10:35:50.622626+0000","last_fullsized":"2024-03-24T10:35:50.622626+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635400+0000","last_change":"2024-03-24T10:35:50.631769+0000","last_active":"2024-03-24T10:35:50.635400+0000","last_peered":"2024-03-24T10:35:50.635400+0000","last_clean":"2024-03-24T10:35:50.635400+0000","last_became_active":"2024-03-24T10:35:50.631453+0000","last_became_peered":"2024-03-24T10:35:50.631453+0000","last_unstale":"2024-03-24T10:35:50.635400+0000","last_undegraded":"2024-03-24T10:35:50.635400+0000","last_fullsized":"2024-03-24T10:35:50.635400+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:11.009937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626484+0000","last_change":"2024-03-24T10:35:50.626484+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626484+0000","last_undegraded":"2024-03-24T10:35:50.626484+0000","last_fullsized":"2024-03-24T10:35:50.626484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638457+0000","last_change":"2024-03-24T10:35:50.636812+0000","last_active":"2024-03-24T10:35:50.638457+0000","last_peered":"2024-03-24T10:35:50.638457+0000","last_clean":"2024-03-24T10:35:50.638457+0000","last_became_active":"2024-03-24T10:35:50.636520+0000","last_became_peered":"2024-03-24T10:35:50.636520+0000","last_unstale":"2024-03-24T10:35:50.638457+0000","last_undegraded":"2024-03-24T10:35:50.638457+0000","last_fullsized":"2024-03-24T10:35:50.638457+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:16.151719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.622920+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:50.622920+0000","last_peered":"2024-03-24T10:35:50.622920+0000","last_clean":"2024-03-24T10:35:50.622920+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:50.622920+0000","last_undegraded":"2024-03-24T10:35:50.622920+0000","last_fullsized":"2024-03-24T10:35:50.622920+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":227616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:35:55.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-24T10:35:55.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619572+0000', 'last_change': '2024-03-24T10:35:50.619572+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619572+0000', 'last_undegraded': '2024-03-24T10:35:50.619572+0000', 'last_fullsized': '2024-03-24T10:35:50.619572+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.468 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-24T10:35:55.468 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622464+0000', 'last_change': '2024-03-24T10:35:50.622464+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622464+0000', 'last_undegraded': '2024-03-24T10:35:50.622464+0000', 'last_fullsized': '2024-03-24T10:35:50.622464+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.627689+0000', 'last_change': '2024-03-24T10:35:50.627689+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.627689+0000', 'last_undegraded': '2024-03-24T10:35:50.627689+0000', 'last_fullsized': '2024-03-24T10:35:50.627689+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621316+0000', 'last_change': '2024-03-24T10:35:50.621316+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621316+0000', 'last_undegraded': '2024-03-24T10:35:50.621316+0000', 'last_fullsized': '2024-03-24T10:35:50.621316+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624565+0000', 'last_change': '2024-03-24T10:35:50.624565+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624565+0000', 'last_undegraded': '2024-03-24T10:35:50.624565+0000', 'last_fullsized': '2024-03-24T10:35:50.624565+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619946+0000', 'last_change': '2024-03-24T10:35:50.619946+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619946+0000', 'last_undegraded': '2024-03-24T10:35:50.619946+0000', 'last_fullsized': '2024-03-24T10:35:50.619946+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624479+0000', 'last_change': '2024-03-24T10:35:50.624479+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624479+0000', 'last_undegraded': '2024-03-24T10:35:50.624479+0000', 'last_fullsized': '2024-03-24T10:35:50.624479+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624154+0000', 'last_change': '2024-03-24T10:35:50.624154+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624154+0000', 'last_undegraded': '2024-03-24T10:35:50.624154+0000', 'last_fullsized': '2024-03-24T10:35:50.624154+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623889+0000', 'last_change': '2024-03-24T10:35:50.623889+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623889+0000', 'last_undegraded': '2024-03-24T10:35:50.623889+0000', 'last_fullsized': '2024-03-24T10:35:50.623889+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619433+0000', 'last_change': '2024-03-24T10:35:50.619433+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619433+0000', 'last_undegraded': '2024-03-24T10:35:50.619433+0000', 'last_fullsized': '2024-03-24T10:35:50.619433+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619034+0000', 'last_change': '2024-03-24T10:35:50.619034+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619034+0000', 'last_undegraded': '2024-03-24T10:35:50.619034+0000', 'last_fullsized': '2024-03-24T10:35:50.619034+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622027+0000', 'last_change': '2024-03-24T10:35:50.622027+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622027+0000', 'last_undegraded': '2024-03-24T10:35:50.622027+0000', 'last_fullsized': '2024-03-24T10:35:50.622027+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-24T10:35:55.470 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619363+0000', 'last_change': '2024-03-24T10:35:50.619363+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619363+0000', 'last_undegraded': '2024-03-24T10:35:50.619363+0000', 'last_fullsized': '2024-03-24T10:35:50.619363+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624300+0000', 'last_change': '2024-03-24T10:35:50.624300+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624300+0000', 'last_undegraded': '2024-03-24T10:35:50.624300+0000', 'last_fullsized': '2024-03-24T10:35:50.624300+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621221+0000', 'last_change': '2024-03-24T10:35:50.621221+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621221+0000', 'last_undegraded': '2024-03-24T10:35:50.621221+0000', 'last_fullsized': '2024-03-24T10:35:50.621221+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618425+0000', 'last_change': '2024-03-24T10:35:50.618425+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618425+0000', 'last_undegraded': '2024-03-24T10:35:50.618425+0000', 'last_fullsized': '2024-03-24T10:35:50.618425+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.627699+0000', 'last_change': '2024-03-24T10:35:50.627699+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.627699+0000', 'last_undegraded': '2024-03-24T10:35:50.627699+0000', 'last_fullsized': '2024-03-24T10:35:50.627699+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622054+0000', 'last_change': '2024-03-24T10:35:50.622054+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622054+0000', 'last_undegraded': '2024-03-24T10:35:50.622054+0000', 'last_fullsized': '2024-03-24T10:35:50.622054+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.627222+0000', 'last_change': '2024-03-24T10:35:50.627222+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.627222+0000', 'last_undegraded': '2024-03-24T10:35:50.627222+0000', 'last_fullsized': '2024-03-24T10:35:50.627222+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.471 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.627419+0000', 'last_change': '2024-03-24T10:35:50.627419+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.627419+0000', 'last_undegraded': '2024-03-24T10:35:50.627419+0000', 'last_fullsized': '2024-03-24T10:35:50.627419+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625486+0000', 'last_change': '2024-03-24T10:35:50.625486+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625486+0000', 'last_undegraded': '2024-03-24T10:35:50.625486+0000', 'last_fullsized': '2024-03-24T10:35:50.625486+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622677+0000', 'last_change': '2024-03-24T10:35:50.622677+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622677+0000', 'last_undegraded': '2024-03-24T10:35:50.622677+0000', 'last_fullsized': '2024-03-24T10:35:50.622677+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618841+0000', 'last_change': '2024-03-24T10:35:50.618841+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618841+0000', 'last_undegraded': '2024-03-24T10:35:50.618841+0000', 'last_fullsized': '2024-03-24T10:35:50.618841+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622708+0000', 'last_change': '2024-03-24T10:35:50.622708+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622708+0000', 'last_undegraded': '2024-03-24T10:35:50.622708+0000', 'last_fullsized': '2024-03-24T10:35:50.622708+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625709+0000', 'last_change': '2024-03-24T10:35:50.625709+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625709+0000', 'last_undegraded': '2024-03-24T10:35:50.625709+0000', 'last_fullsized': '2024-03-24T10:35:50.625709+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.472 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-24T10:35:55.473 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625691+0000', 'last_change': '2024-03-24T10:35:50.625691+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625691+0000', 'last_undegraded': '2024-03-24T10:35:50.625691+0000', 'last_fullsized': '2024-03-24T10:35:50.625691+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618448+0000', 'last_change': '2024-03-24T10:35:50.618448+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618448+0000', 'last_undegraded': '2024-03-24T10:35:50.618448+0000', 'last_fullsized': '2024-03-24T10:35:50.618448+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.626365+0000', 'last_change': '2024-03-24T10:35:50.626365+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.626365+0000', 'last_undegraded': '2024-03-24T10:35:50.626365+0000', 'last_fullsized': '2024-03-24T10:35:50.626365+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.630750+0000', 'last_change': '2024-03-24T10:35:50.630750+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.630750+0000', 'last_undegraded': '2024-03-24T10:35:50.630750+0000', 'last_fullsized': '2024-03-24T10:35:50.630750+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621203+0000', 'last_change': '2024-03-24T10:35:50.621203+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621203+0000', 'last_undegraded': '2024-03-24T10:35:50.621203+0000', 'last_fullsized': '2024-03-24T10:35:50.621203+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623326+0000', 'last_change': '2024-03-24T10:35:50.623326+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623326+0000', 'last_undegraded': '2024-03-24T10:35:50.623326+0000', 'last_fullsized': '2024-03-24T10:35:50.623326+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.474 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624597+0000', 'last_change': '2024-03-24T10:35:50.624597+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624597+0000', 'last_undegraded': '2024-03-24T10:35:50.624597+0000', 'last_fullsized': '2024-03-24T10:35:50.624597+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621815+0000', 'last_change': '2024-03-24T10:35:50.621815+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621815+0000', 'last_undegraded': '2024-03-24T10:35:50.621815+0000', 'last_fullsized': '2024-03-24T10:35:50.621815+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622233+0000', 'last_change': '2024-03-24T10:35:50.622233+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622233+0000', 'last_undegraded': '2024-03-24T10:35:50.622233+0000', 'last_fullsized': '2024-03-24T10:35:50.622233+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.626163+0000', 'last_change': '2024-03-24T10:35:50.626163+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.626163+0000', 'last_undegraded': '2024-03-24T10:35:50.626163+0000', 'last_fullsized': '2024-03-24T10:35:50.626163+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625329+0000', 'last_change': '2024-03-24T10:35:50.625329+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625329+0000', 'last_undegraded': '2024-03-24T10:35:50.625329+0000', 'last_fullsized': '2024-03-24T10:35:50.625329+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623984+0000', 'last_change': '2024-03-24T10:35:50.623984+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623984+0000', 'last_undegraded': '2024-03-24T10:35:50.623984+0000', 'last_fullsized': '2024-03-24T10:35:50.623984+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T10:35:55.475 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621207+0000', 'last_change': '2024-03-24T10:35:50.621207+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621207+0000', 'last_undegraded': '2024-03-24T10:35:50.621207+0000', 'last_fullsized': '2024-03-24T10:35:50.621207+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624799+0000', 'last_change': '2024-03-24T10:35:50.624799+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624799+0000', 'last_undegraded': '2024-03-24T10:35:50.624799+0000', 'last_fullsized': '2024-03-24T10:35:50.624799+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.621406+0000', 'last_change': '2024-03-24T10:35:50.621406+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.621406+0000', 'last_undegraded': '2024-03-24T10:35:50.621406+0000', 'last_fullsized': '2024-03-24T10:35:50.621406+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623498+0000', 'last_change': '2024-03-24T10:35:50.623498+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623498+0000', 'last_undegraded': '2024-03-24T10:35:50.623498+0000', 'last_fullsized': '2024-03-24T10:35:50.623498+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622768+0000', 'last_change': '2024-03-24T10:35:50.622768+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622768+0000', 'last_undegraded': '2024-03-24T10:35:50.622768+0000', 'last_fullsized': '2024-03-24T10:35:50.622768+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623833+0000', 'last_change': '2024-03-24T10:35:50.623833+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623833+0000', 'last_undegraded': '2024-03-24T10:35:50.623833+0000', 'last_fullsized': '2024-03-24T10:35:50.623833+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.476 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623884+0000', 'last_change': '2024-03-24T10:35:50.623884+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623884+0000', 'last_undegraded': '2024-03-24T10:35:50.623884+0000', 'last_fullsized': '2024-03-24T10:35:50.623884+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624072+0000', 'last_change': '2024-03-24T10:35:50.624072+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624072+0000', 'last_undegraded': '2024-03-24T10:35:50.624072+0000', 'last_fullsized': '2024-03-24T10:35:50.624072+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625445+0000', 'last_change': '2024-03-24T10:35:50.625445+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625445+0000', 'last_undegraded': '2024-03-24T10:35:50.625445+0000', 'last_fullsized': '2024-03-24T10:35:50.625445+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.625235+0000', 'last_change': '2024-03-24T10:35:50.625235+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.625235+0000', 'last_undegraded': '2024-03-24T10:35:50.625235+0000', 'last_fullsized': '2024-03-24T10:35:50.625235+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.629902+0000', 'last_change': '2024-03-24T10:35:50.629902+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.629902+0000', 'last_undegraded': '2024-03-24T10:35:50.629902+0000', 'last_fullsized': '2024-03-24T10:35:50.629902+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.619769+0000', 'last_change': '2024-03-24T10:35:50.619769+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.619769+0000', 'last_undegraded': '2024-03-24T10:35:50.619769+0000', 'last_fullsized': '2024-03-24T10:35:50.619769+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.477 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618403+0000', 'last_change': '2024-03-24T10:35:50.618403+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618403+0000', 'last_undegraded': '2024-03-24T10:35:50.618403+0000', 'last_fullsized': '2024-03-24T10:35:50.618403+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624501+0000', 'last_change': '2024-03-24T10:35:50.624501+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624501+0000', 'last_undegraded': '2024-03-24T10:35:50.624501+0000', 'last_fullsized': '2024-03-24T10:35:50.624501+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618657+0000', 'last_change': '2024-03-24T10:35:50.618657+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618657+0000', 'last_undegraded': '2024-03-24T10:35:50.618657+0000', 'last_fullsized': '2024-03-24T10:35:50.618657+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623651+0000', 'last_change': '2024-03-24T10:35:50.623651+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623651+0000', 'last_undegraded': '2024-03-24T10:35:50.623651+0000', 'last_fullsized': '2024-03-24T10:35:50.623651+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.628548+0000', 'last_change': '2024-03-24T10:35:50.628548+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.628548+0000', 'last_undegraded': '2024-03-24T10:35:50.628548+0000', 'last_fullsized': '2024-03-24T10:35:50.628548+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-24T10:35:55.478 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.626260+0000', 'last_change': '2024-03-24T10:35:50.626260+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.626260+0000', 'last_undegraded': '2024-03-24T10:35:50.626260+0000', 'last_fullsized': '2024-03-24T10:35:50.626260+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618384+0000', 'last_change': '2024-03-24T10:35:50.618384+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618384+0000', 'last_undegraded': '2024-03-24T10:35:50.618384+0000', 'last_fullsized': '2024-03-24T10:35:50.618384+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623104+0000', 'last_change': '2024-03-24T10:35:50.623104+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623104+0000', 'last_undegraded': '2024-03-24T10:35:50.623104+0000', 'last_fullsized': '2024-03-24T10:35:50.623104+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.618732+0000', 'last_change': '2024-03-24T10:35:50.618732+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.618732+0000', 'last_undegraded': '2024-03-24T10:35:50.618732+0000', 'last_fullsized': '2024-03-24T10:35:50.618732+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624509+0000', 'last_change': '2024-03-24T10:35:50.624509+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624509+0000', 'last_undegraded': '2024-03-24T10:35:50.624509+0000', 'last_fullsized': '2024-03-24T10:35:50.624509+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.624517+0000', 'last_change': '2024-03-24T10:35:50.624517+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.624517+0000', 'last_undegraded': '2024-03-24T10:35:50.624517+0000', 'last_fullsized': '2024-03-24T10:35:50.624517+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-24T10:35:55.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.623387+0000', 'last_change': '2024-03-24T10:35:50.623387+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.623387+0000', 'last_undegraded': '2024-03-24T10:35:50.623387+0000', 'last_fullsized': '2024-03-24T10:35:50.623387+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.630039+0000', 'last_change': '2024-03-24T10:35:50.630039+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.630039+0000', 'last_undegraded': '2024-03-24T10:35:50.630039+0000', 'last_fullsized': '2024-03-24T10:35:50.630039+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.622626+0000', 'last_change': '2024-03-24T10:35:50.622626+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.622626+0000', 'last_undegraded': '2024-03-24T10:35:50.622626+0000', 'last_fullsized': '2024-03-24T10:35:50.622626+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-24T10:35:55.480 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 48, 'state': 'creating+peering', 'last_fresh': '2024-03-24T10:35:50.626484+0000', 'last_change': '2024-03-24T10:35:50.626484+0000', 'last_active': '2024-03-24T10:35:50.608329+0000', 'last_peered': '2024-03-24T10:35:50.608329+0000', 'last_clean': '2024-03-24T10:35:50.608329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T10:35:50.626484+0000', 'last_undegraded': '2024-03-24T10:35:50.626484+0000', 'last_fullsized': '2024-03-24T10:35:50.626484+0000', 'mapping_epoch': 48, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 48, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'last_clean_scrub_stamp': '2024-03-24T10:35:50.608329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T10:35:55.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T10:35:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.609+0000 7f601de9d640 1 Processor -- start 2024-03-24T10:35:55.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.609+0000 7f601de9d640 1 -- start start 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f601de9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f6018166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f601de9d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6018071300 con 0x7f60180cec10 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f601de9d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601806d2a0 con 0x7f6018145690 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f601de9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60180cf050 con 0x7f6018071ea0 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f6016ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60180cec10 0x7f6018071790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49930/0 (socket says 172.21.15.1:49930) 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f6016ffd640 1 -- 172.21.15.1:0/2701661020 learned_addr learned my addr 172.21.15.1:0/2701661020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f6017fff640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f6018166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740828821 0 0) 0x7f6018071300 con 0x7f60180cec10 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6000003520 con 0x7f60180cec10 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1313539061 0 0) 0x7f6000003520 con 0x7f60180cec10 2024-03-24T10:35:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 msgr2=0x7f6018166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f6018166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 >> v1:172.21.15.3:6789/0 conn(0x7f6018145690 legacy=0x7f601806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6018145a70 con 0x7f60180cec10 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.610+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f600c001670 con 0x7f60180cec10 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 >> v1:172.21.15.1:6789/0 conn(0x7f60180cec10 legacy=0x7f6018071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60000039a0 0x7f6000003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f60000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6000004d00 0x7f6000005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f60180cf050 con 0x7f6000004d00 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f600000ba30 con 0x7f60000039a0 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f600000bbf0 con 0x7f6018071ea0 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6000004d00 0x7f6000005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6016ffd640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f60000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60177fe640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60000039a0 0x7f6000003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 -- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 msgr2=0x7f60000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018071ea0 0x7f60000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 -- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60000039a0 msgr2=0x7f6000003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60000039a0 0x7f6000003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f6017fff640 1 -- 172.21.15.1:0/2701661020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6000003520 con 0x7f6000004d00 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.611+0000 7f60177fe640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60000039a0 0x7f6000003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f6017fff640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6000004d00 0x7f6000005180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6010009f90 tx=0x7f60000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3642b2e550d3e1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601002cbd0 con 0x7f6000004d00 2024-03-24T10:35:55.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f601000f040 con 0x7f6000004d00 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f60167fc640 1 -- 172.21.15.1:0/2701661020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010034440 con 0x7f6000004d00 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f601de9d640 1 -- 172.21.15.1:0/2701661020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6000004d00 msgr2=0x7f6000005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f601de9d640 1 --2- 172.21.15.1:0/2701661020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6000004d00 0x7f6000005180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6010009f90 tx=0x7f60000057f0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.612+0000 7f6017fff640 1 -- 172.21.15.1:0/2701661020 reap_dead start 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.613+0000 7f601de9d640 1 -- 172.21.15.1:0/2701661020 shutdown_connections 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.613+0000 7f601de9d640 1 -- 172.21.15.1:0/2701661020 >> 172.21.15.1:0/2701661020 conn(0x7f60180b1710 msgr2=0x7f60180b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.613+0000 7f601de9d640 1 -- 172.21.15.1:0/2701661020 shutdown_connections 2024-03-24T10:35:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.613+0000 7f601de9d640 1 -- 172.21.15.1:0/2701661020 wait complete. 2024-03-24T10:35:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.614+0000 7f601de9d640 1 Processor -- start 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.614+0000 7f601de9d640 1 -- start start 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180aed40 0x7f6018161420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018161960 0x7f60181f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f60181f48e0 con 0x7f60180af120 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f60181f4320 con 0x7f60180aed40 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f60181f4600 con 0x7f6018161960 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38644/0 (socket says 172.21.15.1:38644) 2024-03-24T10:35:55.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 -- 172.21.15.1:0/4225499344 learned_addr learned my addr 172.21.15.1:0/4225499344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6016ffd640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018161960 0x7f60181f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f60177fe640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180aed40 0x7f6018161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 -- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018161960 msgr2=0x7f60181f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018161960 0x7f60181f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 -- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180aed40 msgr2=0x7f6018161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180aed40 0x7f6018161420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 -- 172.21.15.1:0/4225499344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6018160c40 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f6017fff640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6010009f90 tx=0x7f6010009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010043070 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f601002ce50 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6018160f20 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.615+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010034d90 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.616+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60180adb70 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.617+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6018160aa0 con 0x7f60180af120 2024-03-24T10:35:55.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.617+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f601003e440 con 0x7f60180af120 2024-03-24T10:35:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.617+0000 7f5ffffff640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.618+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f60100abb50 con 0x7f60180af120 2024-03-24T10:35:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.618+0000 7f60177fe640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:55.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.618+0000 7f60177fe640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 0x7f600406dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6008007fd0 tx=0x7f6008009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:55.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.622+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6018160aa0 con 0x7f60180af120 2024-03-24T10:35:55.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.874+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6018160aa0 con 0x7f60180af120 2024-03-24T10:35:55.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.874+0000 7f5ffffff640 1 -- 172.21.15.1:0/4225499344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1453 (secure 0 0 0) 0x7f6018160aa0 con 0x7f60180af120 2024-03-24T10:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:55.875 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":145,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":50,"num_osds":8,"num_up_osds":8,"osd_up_since":1711276421,"num_in_osds":8,"osd_in_since":1711276412,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":244146176,"bytes_avail":772849967104,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":25,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5982}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-24T10:35:32.535458+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 msgr2=0x7f600406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 0x7f600406dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6008007fd0 tx=0x7f6008009040 comp rx=0 tx=0).stop 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 msgr2=0x7f601815e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6010009f90 tx=0x7f6010009750 comp rx=0 tx=0).stop 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 shutdown_connections 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60180af120 0x7f601815e8f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6018161960 0x7f60181f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 --2- 172.21.15.1:0/4225499344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180aed40 0x7f6018161420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:55.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 >> 172.21.15.1:0/4225499344 conn(0x7f60180b1710 msgr2=0x7f6018164140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:55.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.876+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 shutdown_connections 2024-03-24T10:35:55.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:55.877+0000 7f601de9d640 1 -- 172.21.15.1:0/4225499344 wait complete. 2024-03-24T10:35:55.887 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T10:35:55.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:35:56.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.022+0000 7f11db53a640 1 Processor -- start 2024-03-24T10:35:56.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 -- start start 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11d415e390 0x7f11d40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40d1000 0x7f11d40d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d415e770 con 0x7f11d40b3320 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f11d40b3190 con 0x7f11d415e390 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11db53a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f11d40d4e40 con 0x7f11d40d1000 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11d92d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11d415e390 0x7f11d40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11d92d6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11d415e390 0x7f11d40d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38656/0 (socket says 172.21.15.1:38656) 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.023+0000 7f11d92d6640 1 -- 172.21.15.1:0/1009465751 learned_addr learned my addr 172.21.15.1:0/1009465751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.024+0000 7f11d8ad5640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40d1000 0x7f11d40d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.024+0000 7f11d92d6640 1 -- 172.21.15.1:0/1009465751 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40d1000 msgr2=0x7f11d40d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.024+0000 7f11d92d6640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40d1000 0x7f11d40d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.024+0000 7f11d92d6640 1 -- 172.21.15.1:0/1009465751 >> v1:172.21.15.3:6789/0 conn(0x7f11d40b3320 legacy=0x7f11d40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.024+0000 7f11d92d6640 1 -- 172.21.15.1:0/1009465751 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d40d1e40 con 0x7f11d415e390 2024-03-24T10:35:56.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11d92d6640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d24a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009e50 tx=0x7f11cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fa3ae29e7a446f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:56.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11d3fff640 1 -- 172.21.15.1:0/1009465751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc02c9e0 con 0x7f11d415e390 2024-03-24T10:35:56.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11d3fff640 1 -- 172.21.15.1:0/1009465751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11cc02cb40 con 0x7f11d415e390 2024-03-24T10:35:56.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11d3fff640 1 -- 172.21.15.1:0/1009465751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc0343f0 con 0x7f11d415e390 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 -- 172.21.15.1:0/1009465751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 msgr2=0x7f11d40d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d24a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009e50 tx=0x7f11cc02be40 comp rx=0 tx=0).stop 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 -- 172.21.15.1:0/1009465751 shutdown_connections 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40d1000 0x7f11d40d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 --2- 172.21.15.1:0/1009465751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d24a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 -- 172.21.15.1:0/1009465751 >> 172.21.15.1:0/1009465751 conn(0x7f11d40d3ec0 msgr2=0x7f11d40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 -- 172.21.15.1:0/1009465751 shutdown_connections 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.025+0000 7f11db53a640 1 -- 172.21.15.1:0/1009465751 wait complete. 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.026+0000 7f11db53a640 1 Processor -- start 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 -- start start 2024-03-24T10:35:56.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40b3320 0x7f11d40cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11d40d1000 0x7f11d415c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40cb7b0 con 0x7f11d415e390 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40cb110 con 0x7f11d40d1000 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11db53a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f11d41f4930 con 0x7f11d40b3320 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11d9ad7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11d9ad7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38662/0 (socket says 172.21.15.1:38662) 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.027+0000 7f11d9ad7640 1 -- 172.21.15.1:0/1492772476 learned_addr learned my addr 172.21.15.1:0/1492772476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d8ad5640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11d40d1000 0x7f11d415c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 -- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40b3320 msgr2=0x7f11d40cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d92d6640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40b3320 0x7f11d40cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40b3320 0x7f11d40cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 -- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11d40d1000 msgr2=0x7f11d415c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11d40d1000 0x7f11d415c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 -- 172.21.15.1:0/1492772476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc009b30 con 0x7f11d415e390 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d9ad7640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c3a0 tx=0x7f11c400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c40052d0 con 0x7f11d415e390 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4005430 con 0x7f11d415e390 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d40cb3d0 con 0x7f11d415e390 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4003830 con 0x7f11d415e390 2024-03-24T10:35:56.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.028+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d40d3310 con 0x7f11d415e390 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.029+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d40cb8f0 con 0x7f11d415e390 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.030+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11c4003990 con 0x7f11d415e390 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.030+0000 7f11d1ffb640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 0x7f11c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.030+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f11c40895e0 con 0x7f11d415e390 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.030+0000 7f11d92d6640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 0x7f11c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:56.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.030+0000 7f11d92d6640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 0x7f11c006dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f11cc035000 tx=0x7f11cc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:56.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.034+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11d40cb8f0 con 0x7f11d415e390 2024-03-24T10:35:56.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:35:56.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:35:56.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.212+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11d40caf90 con 0x7f11c006b810 2024-03-24T10:35:56.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.224+0000 7f11d1ffb640 1 -- 172.21.15.1:0/1492772476 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246641 (secure 0 0 0) 0x7f11cc0343f0 con 0x7f11c006b810 2024-03-24T10:35:56.227 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:56.228 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:35:56.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 msgr2=0x7f11c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 0x7f11c006dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f11cc035000 tx=0x7f11cc030000 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 msgr2=0x7f11d40d2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f11c400c3a0 tx=0x7f11c400c860 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 shutdown_connections 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f11c006b810 0x7f11c006dcd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11d415e390 0x7f11d40d2dd0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11d40d1000 0x7f11d415c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 --2- 172.21.15.1:0/1492772476 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f11d40b3320 0x7f11d40cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 >> 172.21.15.1:0/1492772476 conn(0x7f11d40d3ec0 msgr2=0x7f11d40d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 shutdown_connections 2024-03-24T10:35:56.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:56.227+0000 7f11db53a640 1 -- 172.21.15.1:0/1492772476 wait complete. 2024-03-24T10:35:56.239 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-03-24T10:35:54.539396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238424,"kb_used_data":5596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736296,"statfs":{"total":773094113280,"available":772849967104,"internally_reserved":0,"allocated":5730304,"data_stored":3045589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931065"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619572+0000","last_change":"2024-03-24T10:35:50.619572+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619572+0000","last_undegraded":"2024-03-24T10:35:50.619572+0000","last_fullsized":"2024-03-24T10:35:50.619572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622464+0000","last_change":"2024-03-24T10:35:50.622464+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622464+0000","last_undegraded":"2024-03-24T10:35:50.622464+0000","last_fullsized":"2024-03-24T10:35:50.622464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627689+0000","last_change":"2024-03-24T10:35:50.627689+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627689+0000","last_undegraded":"2024-03-24T10:35:50.627689+0000","last_fullsized":"2024-03-24T10:35:50.627689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621316+0000","last_change":"2024-03-24T10:35:50.621316+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621316+0000","last_undegraded":"2024-03-24T10:35:50.621316+0000","last_fullsized":"2024-03-24T10:35:50.621316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624565+0000","last_change":"2024-03-24T10:35:50.624565+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624565+0000","last_undegraded":"2024-03-24T10:35:50.624565+0000","last_fullsized":"2024-03-24T10:35:50.624565+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619946+0000","last_change":"2024-03-24T10:35:50.619946+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619946+0000","last_undegraded":"2024-03-24T10:35:50.619946+0000","last_fullsized":"2024-03-24T10:35:50.619946+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624479+0000","last_change":"2024-03-24T10:35:50.624479+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624479+0000","last_undegraded":"2024-03-24T10:35:50.624479+0000","last_fullsized":"2024-03-24T10:35:50.624479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624154+0000","last_change":"2024-03-24T10:35:50.624154+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624154+0000","last_undegraded":"2024-03-24T10:35:50.624154+0000","last_fullsized":"2024-03-24T10:35:50.624154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623889+0000","last_change":"2024-03-24T10:35:50.623889+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623889+0000","last_undegraded":"2024-03-24T10:35:50.623889+0000","last_fullsized":"2024-03-24T10:35:50.623889+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619433+0000","last_change":"2024-03-24T10:35:50.619433+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619433+0000","last_undegraded":"2024-03-24T10:35:50.619433+0000","last_fullsized":"2024-03-24T10:35:50.619433+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619034+0000","last_change":"2024-03-24T10:35:50.619034+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619034+0000","last_undegraded":"2024-03-24T10:35:50.619034+0000","last_fullsized":"2024-03-24T10:35:50.619034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622027+0000","last_change":"2024-03-24T10:35:50.622027+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622027+0000","last_undegraded":"2024-03-24T10:35:50.622027+0000","last_fullsized":"2024-03-24T10:35:50.622027+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619363+0000","last_change":"2024-03-24T10:35:50.619363+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619363+0000","last_undegraded":"2024-03-24T10:35:50.619363+0000","last_fullsized":"2024-03-24T10:35:50.619363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624300+0000","last_change":"2024-03-24T10:35:50.624300+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624300+0000","last_undegraded":"2024-03-24T10:35:50.624300+0000","last_fullsized":"2024-03-24T10:35:50.624300+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621221+0000","last_change":"2024-03-24T10:35:50.621221+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621221+0000","last_undegraded":"2024-03-24T10:35:50.621221+0000","last_fullsized":"2024-03-24T10:35:50.621221+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618425+0000","last_change":"2024-03-24T10:35:50.618425+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618425+0000","last_undegraded":"2024-03-24T10:35:50.618425+0000","last_fullsized":"2024-03-24T10:35:50.618425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627699+0000","last_change":"2024-03-24T10:35:50.627699+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627699+0000","last_undegraded":"2024-03-24T10:35:50.627699+0000","last_fullsized":"2024-03-24T10:35:50.627699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622054+0000","last_change":"2024-03-24T10:35:50.622054+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622054+0000","last_undegraded":"2024-03-24T10:35:50.622054+0000","last_fullsized":"2024-03-24T10:35:50.622054+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627222+0000","last_change":"2024-03-24T10:35:50.627222+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627222+0000","last_undegraded":"2024-03-24T10:35:50.627222+0000","last_fullsized":"2024-03-24T10:35:50.627222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.627419+0000","last_change":"2024-03-24T10:35:50.627419+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.627419+0000","last_undegraded":"2024-03-24T10:35:50.627419+0000","last_fullsized":"2024-03-24T10:35:50.627419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625486+0000","last_change":"2024-03-24T10:35:50.625486+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625486+0000","last_undegraded":"2024-03-24T10:35:50.625486+0000","last_fullsized":"2024-03-24T10:35:50.625486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622677+0000","last_change":"2024-03-24T10:35:50.622677+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622677+0000","last_undegraded":"2024-03-24T10:35:50.622677+0000","last_fullsized":"2024-03-24T10:35:50.622677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618841+0000","last_change":"2024-03-24T10:35:50.618841+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618841+0000","last_undegraded":"2024-03-24T10:35:50.618841+0000","last_fullsized":"2024-03-24T10:35:50.618841+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622708+0000","last_change":"2024-03-24T10:35:50.622708+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622708+0000","last_undegraded":"2024-03-24T10:35:50.622708+0000","last_fullsized":"2024-03-24T10:35:50.622708+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625709+0000","last_change":"2024-03-24T10:35:50.625709+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625709+0000","last_undegraded":"2024-03-24T10:35:50.625709+0000","last_fullsized":"2024-03-24T10:35:50.625709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625691+0000","last_change":"2024-03-24T10:35:50.625691+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625691+0000","last_undegraded":"2024-03-24T10:35:50.625691+0000","last_fullsized":"2024-03-24T10:35:50.625691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618448+0000","last_change":"2024-03-24T10:35:50.618448+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618448+0000","last_undegraded":"2024-03-24T10:35:50.618448+0000","last_fullsized":"2024-03-24T10:35:50.618448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626365+0000","last_change":"2024-03-24T10:35:50.626365+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626365+0000","last_undegraded":"2024-03-24T10:35:50.626365+0000","last_fullsized":"2024-03-24T10:35:50.626365+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630750+0000","last_change":"2024-03-24T10:35:50.630750+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630750+0000","last_undegraded":"2024-03-24T10:35:50.630750+0000","last_fullsized":"2024-03-24T10:35:50.630750+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621203+0000","last_change":"2024-03-24T10:35:50.621203+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621203+0000","last_undegraded":"2024-03-24T10:35:50.621203+0000","last_fullsized":"2024-03-24T10:35:50.621203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623326+0000","last_change":"2024-03-24T10:35:50.623326+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623326+0000","last_undegraded":"2024-03-24T10:35:50.623326+0000","last_fullsized":"2024-03-24T10:35:50.623326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624597+0000","last_change":"2024-03-24T10:35:50.624597+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624597+0000","last_undegraded":"2024-03-24T10:35:50.624597+0000","last_fullsized":"2024-03-24T10:35:50.624597+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621815+0000","last_change":"2024-03-24T10:35:50.621815+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621815+0000","last_undegraded":"2024-03-24T10:35:50.621815+0000","last_fullsized":"2024-03-24T10:35:50.621815+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638164+0000","last_change":"2024-03-24T10:35:50.635274+0000","last_active":"2024-03-24T10:35:50.638164+0000","last_peered":"2024-03-24T10:35:50.638164+0000","last_clean":"2024-03-24T10:35:50.638164+0000","last_became_active":"2024-03-24T10:35:50.634934+0000","last_became_peered":"2024-03-24T10:35:50.634934+0000","last_unstale":"2024-03-24T10:35:50.638164+0000","last_undegraded":"2024-03-24T10:35:50.638164+0000","last_fullsized":"2024-03-24T10:35:50.638164+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:19.768286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622233+0000","last_change":"2024-03-24T10:35:50.622233+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622233+0000","last_undegraded":"2024-03-24T10:35:50.622233+0000","last_fullsized":"2024-03-24T10:35:50.622233+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634816+0000","last_change":"2024-03-24T10:35:50.633891+0000","last_active":"2024-03-24T10:35:50.634816+0000","last_peered":"2024-03-24T10:35:50.634816+0000","last_clean":"2024-03-24T10:35:50.634816+0000","last_became_active":"2024-03-24T10:35:50.633604+0000","last_became_peered":"2024-03-24T10:35:50.633604+0000","last_unstale":"2024-03-24T10:35:50.634816+0000","last_undegraded":"2024-03-24T10:35:50.634816+0000","last_fullsized":"2024-03-24T10:35:50.634816+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:45.059275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626163+0000","last_change":"2024-03-24T10:35:50.626163+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626163+0000","last_undegraded":"2024-03-24T10:35:50.626163+0000","last_fullsized":"2024-03-24T10:35:50.626163+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638820+0000","last_change":"2024-03-24T10:35:50.637492+0000","last_active":"2024-03-24T10:35:50.638820+0000","last_peered":"2024-03-24T10:35:50.638820+0000","last_clean":"2024-03-24T10:35:50.638820+0000","last_became_active":"2024-03-24T10:35:50.637225+0000","last_became_peered":"2024-03-24T10:35:50.637225+0000","last_unstale":"2024-03-24T10:35:50.638820+0000","last_undegraded":"2024-03-24T10:35:50.638820+0000","last_fullsized":"2024-03-24T10:35:50.638820+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:12.007000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625329+0000","last_change":"2024-03-24T10:35:50.625329+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625329+0000","last_undegraded":"2024-03-24T10:35:50.625329+0000","last_fullsized":"2024-03-24T10:35:50.625329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638326+0000","last_change":"2024-03-24T10:35:50.632990+0000","last_active":"2024-03-24T10:35:50.638326+0000","last_peered":"2024-03-24T10:35:50.638326+0000","last_clean":"2024-03-24T10:35:50.638326+0000","last_became_active":"2024-03-24T10:35:50.632635+0000","last_became_peered":"2024-03-24T10:35:50.632635+0000","last_unstale":"2024-03-24T10:35:50.638326+0000","last_undegraded":"2024-03-24T10:35:50.638326+0000","last_fullsized":"2024-03-24T10:35:50.638326+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:47.712468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623984+0000","last_change":"2024-03-24T10:35:50.623984+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623984+0000","last_undegraded":"2024-03-24T10:35:50.623984+0000","last_fullsized":"2024-03-24T10:35:50.623984+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639222+0000","last_change":"2024-03-24T10:35:50.638378+0000","last_active":"2024-03-24T10:35:50.639222+0000","last_peered":"2024-03-24T10:35:50.639222+0000","last_clean":"2024-03-24T10:35:50.639222+0000","last_became_active":"2024-03-24T10:35:50.638077+0000","last_became_peered":"2024-03-24T10:35:50.638077+0000","last_unstale":"2024-03-24T10:35:50.639222+0000","last_undegraded":"2024-03-24T10:35:50.639222+0000","last_fullsized":"2024-03-24T10:35:50.639222+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:48:48.966513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621207+0000","last_change":"2024-03-24T10:35:50.621207+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621207+0000","last_undegraded":"2024-03-24T10:35:50.621207+0000","last_fullsized":"2024-03-24T10:35:50.621207+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638629+0000","last_change":"2024-03-24T10:35:50.636470+0000","last_active":"2024-03-24T10:35:50.638629+0000","last_peered":"2024-03-24T10:35:50.638629+0000","last_clean":"2024-03-24T10:35:50.638629+0000","last_became_active":"2024-03-24T10:35:50.636139+0000","last_became_peered":"2024-03-24T10:35:50.636139+0000","last_unstale":"2024-03-24T10:35:50.638629+0000","last_undegraded":"2024-03-24T10:35:50.638629+0000","last_fullsized":"2024-03-24T10:35:50.638629+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:55.032277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624799+0000","last_change":"2024-03-24T10:35:50.624799+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624799+0000","last_undegraded":"2024-03-24T10:35:50.624799+0000","last_fullsized":"2024-03-24T10:35:50.624799+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637837+0000","last_change":"2024-03-24T10:35:50.635522+0000","last_active":"2024-03-24T10:35:50.637837+0000","last_peered":"2024-03-24T10:35:50.637837+0000","last_clean":"2024-03-24T10:35:50.637837+0000","last_became_active":"2024-03-24T10:35:50.635036+0000","last_became_peered":"2024-03-24T10:35:50.635036+0000","last_unstale":"2024-03-24T10:35:50.637837+0000","last_undegraded":"2024-03-24T10:35:50.637837+0000","last_fullsized":"2024-03-24T10:35:50.637837+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:55.117568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.621406+0000","last_change":"2024-03-24T10:35:50.621406+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.621406+0000","last_undegraded":"2024-03-24T10:35:50.621406+0000","last_fullsized":"2024-03-24T10:35:50.621406+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638491+0000","last_change":"2024-03-24T10:35:50.636110+0000","last_active":"2024-03-24T10:35:50.638491+0000","last_peered":"2024-03-24T10:35:50.638491+0000","last_clean":"2024-03-24T10:35:50.638491+0000","last_became_active":"2024-03-24T10:35:50.635804+0000","last_became_peered":"2024-03-24T10:35:50.635804+0000","last_unstale":"2024-03-24T10:35:50.638491+0000","last_undegraded":"2024-03-24T10:35:50.638491+0000","last_fullsized":"2024-03-24T10:35:50.638491+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:26.730054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623498+0000","last_change":"2024-03-24T10:35:50.623498+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623498+0000","last_undegraded":"2024-03-24T10:35:50.623498+0000","last_fullsized":"2024-03-24T10:35:50.623498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634675+0000","last_change":"2024-03-24T10:35:50.630099+0000","last_active":"2024-03-24T10:35:50.634675+0000","last_peered":"2024-03-24T10:35:50.634675+0000","last_clean":"2024-03-24T10:35:50.634675+0000","last_became_active":"2024-03-24T10:35:50.629665+0000","last_became_peered":"2024-03-24T10:35:50.629665+0000","last_unstale":"2024-03-24T10:35:50.634675+0000","last_undegraded":"2024-03-24T10:35:50.634675+0000","last_fullsized":"2024-03-24T10:35:50.634675+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:10.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622768+0000","last_change":"2024-03-24T10:35:50.622768+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622768+0000","last_undegraded":"2024-03-24T10:35:50.622768+0000","last_fullsized":"2024-03-24T10:35:50.622768+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639380+0000","last_change":"2024-03-24T10:35:50.638305+0000","last_active":"2024-03-24T10:35:50.639380+0000","last_peered":"2024-03-24T10:35:50.639380+0000","last_clean":"2024-03-24T10:35:50.639380+0000","last_became_active":"2024-03-24T10:35:50.637768+0000","last_became_peered":"2024-03-24T10:35:50.637768+0000","last_unstale":"2024-03-24T10:35:50.639380+0000","last_undegraded":"2024-03-24T10:35:50.639380+0000","last_fullsized":"2024-03-24T10:35:50.639380+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:33:50.611475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623833+0000","last_change":"2024-03-24T10:35:50.623833+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623833+0000","last_undegraded":"2024-03-24T10:35:50.623833+0000","last_fullsized":"2024-03-24T10:35:50.623833+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634930+0000","last_change":"2024-03-24T10:35:50.631873+0000","last_active":"2024-03-24T10:35:50.634930+0000","last_peered":"2024-03-24T10:35:50.634930+0000","last_clean":"2024-03-24T10:35:50.634930+0000","last_became_active":"2024-03-24T10:35:50.631567+0000","last_became_peered":"2024-03-24T10:35:50.631567+0000","last_unstale":"2024-03-24T10:35:50.634930+0000","last_undegraded":"2024-03-24T10:35:50.634930+0000","last_fullsized":"2024-03-24T10:35:50.634930+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:29.632034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623884+0000","last_change":"2024-03-24T10:35:50.623884+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623884+0000","last_undegraded":"2024-03-24T10:35:50.623884+0000","last_fullsized":"2024-03-24T10:35:50.623884+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632135+0000","last_change":"2024-03-24T10:35:50.626864+0000","last_active":"2024-03-24T10:35:50.632135+0000","last_peered":"2024-03-24T10:35:50.632135+0000","last_clean":"2024-03-24T10:35:50.632135+0000","last_became_active":"2024-03-24T10:35:50.626556+0000","last_became_peered":"2024-03-24T10:35:50.626556+0000","last_unstale":"2024-03-24T10:35:50.632135+0000","last_undegraded":"2024-03-24T10:35:50.632135+0000","last_fullsized":"2024-03-24T10:35:50.632135+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:40.363003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624072+0000","last_change":"2024-03-24T10:35:50.624072+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624072+0000","last_undegraded":"2024-03-24T10:35:50.624072+0000","last_fullsized":"2024-03-24T10:35:50.624072+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638533+0000","last_change":"2024-03-24T10:35:50.637003+0000","last_active":"2024-03-24T10:35:50.638533+0000","last_peered":"2024-03-24T10:35:50.638533+0000","last_clean":"2024-03-24T10:35:50.638533+0000","last_became_active":"2024-03-24T10:35:50.636716+0000","last_became_peered":"2024-03-24T10:35:50.636716+0000","last_unstale":"2024-03-24T10:35:50.638533+0000","last_undegraded":"2024-03-24T10:35:50.638533+0000","last_fullsized":"2024-03-24T10:35:50.638533+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:13.161270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625445+0000","last_change":"2024-03-24T10:35:50.625445+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625445+0000","last_undegraded":"2024-03-24T10:35:50.625445+0000","last_fullsized":"2024-03-24T10:35:50.625445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638748+0000","last_change":"2024-03-24T10:35:50.637095+0000","last_active":"2024-03-24T10:35:50.638748+0000","last_peered":"2024-03-24T10:35:50.638748+0000","last_clean":"2024-03-24T10:35:50.638748+0000","last_became_active":"2024-03-24T10:35:50.636717+0000","last_became_peered":"2024-03-24T10:35:50.636717+0000","last_unstale":"2024-03-24T10:35:50.638748+0000","last_undegraded":"2024-03-24T10:35:50.638748+0000","last_fullsized":"2024-03-24T10:35:50.638748+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.035067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.625235+0000","last_change":"2024-03-24T10:35:50.625235+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.625235+0000","last_undegraded":"2024-03-24T10:35:50.625235+0000","last_fullsized":"2024-03-24T10:35:50.625235+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638235+0000","last_change":"2024-03-24T10:35:50.637013+0000","last_active":"2024-03-24T10:35:50.638235+0000","last_peered":"2024-03-24T10:35:50.638235+0000","last_clean":"2024-03-24T10:35:50.638235+0000","last_became_active":"2024-03-24T10:35:50.636281+0000","last_became_peered":"2024-03-24T10:35:50.636281+0000","last_unstale":"2024-03-24T10:35:50.638235+0000","last_undegraded":"2024-03-24T10:35:50.638235+0000","last_fullsized":"2024-03-24T10:35:50.638235+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:40.428655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.629902+0000","last_change":"2024-03-24T10:35:50.629902+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.629902+0000","last_undegraded":"2024-03-24T10:35:50.629902+0000","last_fullsized":"2024-03-24T10:35:50.629902+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.630082+0000","last_change":"2024-03-24T10:35:50.626837+0000","last_active":"2024-03-24T10:35:50.630082+0000","last_peered":"2024-03-24T10:35:50.630082+0000","last_clean":"2024-03-24T10:35:50.630082+0000","last_became_active":"2024-03-24T10:35:50.626538+0000","last_became_peered":"2024-03-24T10:35:50.626538+0000","last_unstale":"2024-03-24T10:35:50.630082+0000","last_undegraded":"2024-03-24T10:35:50.630082+0000","last_fullsized":"2024-03-24T10:35:50.630082+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:55.782184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.619769+0000","last_change":"2024-03-24T10:35:50.619769+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.619769+0000","last_undegraded":"2024-03-24T10:35:50.619769+0000","last_fullsized":"2024-03-24T10:35:50.619769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639683+0000","last_change":"2024-03-24T10:35:50.638452+0000","last_active":"2024-03-24T10:35:50.639683+0000","last_peered":"2024-03-24T10:35:50.639683+0000","last_clean":"2024-03-24T10:35:50.639683+0000","last_became_active":"2024-03-24T10:35:50.636353+0000","last_became_peered":"2024-03-24T10:35:50.636353+0000","last_unstale":"2024-03-24T10:35:50.639683+0000","last_undegraded":"2024-03-24T10:35:50.639683+0000","last_fullsized":"2024-03-24T10:35:50.639683+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:20.996736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618403+0000","last_change":"2024-03-24T10:35:50.618403+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618403+0000","last_undegraded":"2024-03-24T10:35:50.618403+0000","last_fullsized":"2024-03-24T10:35:50.618403+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.629210+0000","last_change":"2024-03-24T10:35:50.628002+0000","last_active":"2024-03-24T10:35:50.629210+0000","last_peered":"2024-03-24T10:35:50.629210+0000","last_clean":"2024-03-24T10:35:50.629210+0000","last_became_active":"2024-03-24T10:35:50.627615+0000","last_became_peered":"2024-03-24T10:35:50.627615+0000","last_unstale":"2024-03-24T10:35:50.629210+0000","last_undegraded":"2024-03-24T10:35:50.629210+0000","last_fullsized":"2024-03-24T10:35:50.629210+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:04:01.756032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624501+0000","last_change":"2024-03-24T10:35:50.624501+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624501+0000","last_undegraded":"2024-03-24T10:35:50.624501+0000","last_fullsized":"2024-03-24T10:35:50.624501+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635488+0000","last_change":"2024-03-24T10:35:50.632552+0000","last_active":"2024-03-24T10:35:50.635488+0000","last_peered":"2024-03-24T10:35:50.635488+0000","last_clean":"2024-03-24T10:35:50.635488+0000","last_became_active":"2024-03-24T10:35:50.632243+0000","last_became_peered":"2024-03-24T10:35:50.632243+0000","last_unstale":"2024-03-24T10:35:50.635488+0000","last_undegraded":"2024-03-24T10:35:50.635488+0000","last_fullsized":"2024-03-24T10:35:50.635488+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:44.471627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618657+0000","last_change":"2024-03-24T10:35:50.618657+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618657+0000","last_undegraded":"2024-03-24T10:35:50.618657+0000","last_fullsized":"2024-03-24T10:35:50.618657+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637051+0000","last_change":"2024-03-24T10:35:50.632996+0000","last_active":"2024-03-24T10:35:50.637051+0000","last_peered":"2024-03-24T10:35:50.637051+0000","last_clean":"2024-03-24T10:35:50.637051+0000","last_became_active":"2024-03-24T10:35:50.632669+0000","last_became_peered":"2024-03-24T10:35:50.632669+0000","last_unstale":"2024-03-24T10:35:50.637051+0000","last_undegraded":"2024-03-24T10:35:50.637051+0000","last_fullsized":"2024-03-24T10:35:50.637051+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:45:55.867715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623651+0000","last_change":"2024-03-24T10:35:50.623651+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623651+0000","last_undegraded":"2024-03-24T10:35:50.623651+0000","last_fullsized":"2024-03-24T10:35:50.623651+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.633750+0000","last_change":"2024-03-24T10:35:50.631866+0000","last_active":"2024-03-24T10:35:50.633750+0000","last_peered":"2024-03-24T10:35:50.633750+0000","last_clean":"2024-03-24T10:35:50.633750+0000","last_became_active":"2024-03-24T10:35:50.631578+0000","last_became_peered":"2024-03-24T10:35:50.631578+0000","last_unstale":"2024-03-24T10:35:50.633750+0000","last_undegraded":"2024-03-24T10:35:50.633750+0000","last_fullsized":"2024-03-24T10:35:50.633750+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:57.449362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.628548+0000","last_change":"2024-03-24T10:35:50.628548+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.628548+0000","last_undegraded":"2024-03-24T10:35:50.628548+0000","last_fullsized":"2024-03-24T10:35:50.628548+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632090+0000","last_change":"2024-03-24T10:35:50.629954+0000","last_active":"2024-03-24T10:35:50.632090+0000","last_peered":"2024-03-24T10:35:50.632090+0000","last_clean":"2024-03-24T10:35:50.632090+0000","last_became_active":"2024-03-24T10:35:50.629658+0000","last_became_peered":"2024-03-24T10:35:50.629658+0000","last_unstale":"2024-03-24T10:35:50.632090+0000","last_undegraded":"2024-03-24T10:35:50.632090+0000","last_fullsized":"2024-03-24T10:35:50.632090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:41.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626260+0000","last_change":"2024-03-24T10:35:50.626260+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626260+0000","last_undegraded":"2024-03-24T10:35:50.626260+0000","last_fullsized":"2024-03-24T10:35:50.626260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.637838+0000","last_change":"2024-03-24T10:35:50.635109+0000","last_active":"2024-03-24T10:35:50.637838+0000","last_peered":"2024-03-24T10:35:50.637838+0000","last_clean":"2024-03-24T10:35:50.637838+0000","last_became_active":"2024-03-24T10:35:50.634804+0000","last_became_peered":"2024-03-24T10:35:50.634804+0000","last_unstale":"2024-03-24T10:35:50.637838+0000","last_undegraded":"2024-03-24T10:35:50.637838+0000","last_fullsized":"2024-03-24T10:35:50.637838+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:41.493981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618384+0000","last_change":"2024-03-24T10:35:50.618384+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618384+0000","last_undegraded":"2024-03-24T10:35:50.618384+0000","last_fullsized":"2024-03-24T10:35:50.618384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632261+0000","last_change":"2024-03-24T10:35:50.630807+0000","last_active":"2024-03-24T10:35:50.632261+0000","last_peered":"2024-03-24T10:35:50.632261+0000","last_clean":"2024-03-24T10:35:50.632261+0000","last_became_active":"2024-03-24T10:35:50.630512+0000","last_became_peered":"2024-03-24T10:35:50.630512+0000","last_unstale":"2024-03-24T10:35:50.632261+0000","last_undegraded":"2024-03-24T10:35:50.632261+0000","last_fullsized":"2024-03-24T10:35:50.632261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:55.432206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623104+0000","last_change":"2024-03-24T10:35:50.623104+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623104+0000","last_undegraded":"2024-03-24T10:35:50.623104+0000","last_fullsized":"2024-03-24T10:35:50.623104+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.639574+0000","last_change":"2024-03-24T10:35:50.638566+0000","last_active":"2024-03-24T10:35:50.639574+0000","last_peered":"2024-03-24T10:35:50.639574+0000","last_clean":"2024-03-24T10:35:50.639574+0000","last_became_active":"2024-03-24T10:35:50.638280+0000","last_became_peered":"2024-03-24T10:35:50.638280+0000","last_unstale":"2024-03-24T10:35:50.639574+0000","last_undegraded":"2024-03-24T10:35:50.639574+0000","last_fullsized":"2024-03-24T10:35:50.639574+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:31:29.065867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.618732+0000","last_change":"2024-03-24T10:35:50.618732+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.618732+0000","last_undegraded":"2024-03-24T10:35:50.618732+0000","last_fullsized":"2024-03-24T10:35:50.618732+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634647+0000","last_change":"2024-03-24T10:35:50.631008+0000","last_active":"2024-03-24T10:35:50.634647+0000","last_peered":"2024-03-24T10:35:50.634647+0000","last_clean":"2024-03-24T10:35:50.634647+0000","last_became_active":"2024-03-24T10:35:50.630722+0000","last_became_peered":"2024-03-24T10:35:50.630722+0000","last_unstale":"2024-03-24T10:35:50.634647+0000","last_undegraded":"2024-03-24T10:35:50.634647+0000","last_fullsized":"2024-03-24T10:35:50.634647+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:21.376175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624509+0000","last_change":"2024-03-24T10:35:50.624509+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624509+0000","last_undegraded":"2024-03-24T10:35:50.624509+0000","last_fullsized":"2024-03-24T10:35:50.624509+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638327+0000","last_change":"2024-03-24T10:35:50.636330+0000","last_active":"2024-03-24T10:35:50.638327+0000","last_peered":"2024-03-24T10:35:50.638327+0000","last_clean":"2024-03-24T10:35:50.638327+0000","last_became_active":"2024-03-24T10:35:50.636035+0000","last_became_peered":"2024-03-24T10:35:50.636035+0000","last_unstale":"2024-03-24T10:35:50.638327+0000","last_undegraded":"2024-03-24T10:35:50.638327+0000","last_fullsized":"2024-03-24T10:35:50.638327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:54.961171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.624517+0000","last_change":"2024-03-24T10:35:50.624517+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.624517+0000","last_undegraded":"2024-03-24T10:35:50.624517+0000","last_fullsized":"2024-03-24T10:35:50.624517+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.634211+0000","last_change":"2024-03-24T10:35:50.628958+0000","last_active":"2024-03-24T10:35:50.634211+0000","last_peered":"2024-03-24T10:35:50.634211+0000","last_clean":"2024-03-24T10:35:50.634211+0000","last_became_active":"2024-03-24T10:35:50.628661+0000","last_became_peered":"2024-03-24T10:35:50.628661+0000","last_unstale":"2024-03-24T10:35:50.634211+0000","last_undegraded":"2024-03-24T10:35:50.634211+0000","last_fullsized":"2024-03-24T10:35:50.634211+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:01.210812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.623387+0000","last_change":"2024-03-24T10:35:50.623387+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.623387+0000","last_undegraded":"2024-03-24T10:35:50.623387+0000","last_fullsized":"2024-03-24T10:35:50.623387+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.632067+0000","last_change":"2024-03-24T10:35:50.627406+0000","last_active":"2024-03-24T10:35:50.632067+0000","last_peered":"2024-03-24T10:35:50.632067+0000","last_clean":"2024-03-24T10:35:50.632067+0000","last_became_active":"2024-03-24T10:35:50.627116+0000","last_became_peered":"2024-03-24T10:35:50.627116+0000","last_unstale":"2024-03-24T10:35:50.632067+0000","last_undegraded":"2024-03-24T10:35:50.632067+0000","last_fullsized":"2024-03-24T10:35:50.632067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:44.632873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.630039+0000","last_change":"2024-03-24T10:35:50.630039+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.630039+0000","last_undegraded":"2024-03-24T10:35:50.630039+0000","last_fullsized":"2024-03-24T10:35:50.630039+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638602+0000","last_change":"2024-03-24T10:35:50.635584+0000","last_active":"2024-03-24T10:35:50.638602+0000","last_peered":"2024-03-24T10:35:50.638602+0000","last_clean":"2024-03-24T10:35:50.638602+0000","last_became_active":"2024-03-24T10:35:50.635280+0000","last_became_peered":"2024-03-24T10:35:50.635280+0000","last_unstale":"2024-03-24T10:35:50.638602+0000","last_undegraded":"2024-03-24T10:35:50.638602+0000","last_fullsized":"2024-03-24T10:35:50.638602+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:27:14.345670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.622626+0000","last_change":"2024-03-24T10:35:50.622626+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.622626+0000","last_undegraded":"2024-03-24T10:35:50.622626+0000","last_fullsized":"2024-03-24T10:35:50.622626+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.635400+0000","last_change":"2024-03-24T10:35:50.631769+0000","last_active":"2024-03-24T10:35:50.635400+0000","last_peered":"2024-03-24T10:35:50.635400+0000","last_clean":"2024-03-24T10:35:50.635400+0000","last_became_active":"2024-03-24T10:35:50.631453+0000","last_became_peered":"2024-03-24T10:35:50.631453+0000","last_unstale":"2024-03-24T10:35:50.635400+0000","last_undegraded":"2024-03-24T10:35:50.635400+0000","last_fullsized":"2024-03-24T10:35:50.635400+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:11.009937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-03-24T10:35:50.626484+0000","last_change":"2024-03-24T10:35:50.626484+0000","last_active":"2024-03-24T10:35:50.608329+0000","last_peered":"2024-03-24T10:35:50.608329+0000","last_clean":"2024-03-24T10:35:50.608329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T10:35:50.626484+0000","last_undegraded":"2024-03-24T10:35:50.626484+0000","last_fullsized":"2024-03-24T10:35:50.626484+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.638457+0000","last_change":"2024-03-24T10:35:50.636812+0000","last_active":"2024-03-24T10:35:50.638457+0000","last_peered":"2024-03-24T10:35:50.638457+0000","last_clean":"2024-03-24T10:35:50.638457+0000","last_became_active":"2024-03-24T10:35:50.636520+0000","last_became_peered":"2024-03-24T10:35:50.636520+0000","last_unstale":"2024-03-24T10:35:50.638457+0000","last_undegraded":"2024-03-24T10:35:50.638457+0000","last_fullsized":"2024-03-24T10:35:50.638457+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:16.151719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-24T10:35:50.622920+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:50.622920+0000","last_peered":"2024-03-24T10:35:50.622920+0000","last_clean":"2024-03-24T10:35:50.622920+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:50.622920+0000","last_undegraded":"2024-03-24T10:35:50.622920+0000","last_fullsized":"2024-03-24T10:35:50.622920+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":227616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1163264,"data_stored":824259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:35:56.600 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-24T10:35:56.600+0000 7f1a4e90f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:35:59.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:35:59.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.347+0000 7f6613c84640 1 Processor -- start 2024-03-24T10:35:59.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.347+0000 7f6613c84640 1 -- start start 2024-03-24T10:35:59.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.347+0000 7f6613c84640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f660c15e390 0x7f660c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.347+0000 7f6613c84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f660c0d1000 0x7f660c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6613c84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c15e770 con 0x7f660c0b3320 2024-03-24T10:35:59.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6613c84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f660c0b3190 con 0x7f660c0d1000 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6613c84640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f660c0d4e40 con 0x7f660c15e390 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f660c0d1000 0x7f660c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6612221640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f660c0b3320 0x7f660c0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49944/0 (socket says 172.21.15.1:49944) 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f660c0d1000 0x7f660c0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38664/0 (socket says 172.21.15.1:38664) 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6612221640 1 -- 172.21.15.1:0/2365421433 learned_addr learned my addr 172.21.15.1:0/2365421433 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6611a20640 1 --2- 172.21.15.1:0/2365421433 >> v2:172.21.15.3:3300/0 conn(0x7f660c15e390 0x7f660c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 -- 172.21.15.1:0/2365421433 >> v2:172.21.15.3:3300/0 conn(0x7f660c15e390 msgr2=0x7f660c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 --2- 172.21.15.1:0/2365421433 >> v2:172.21.15.3:3300/0 conn(0x7f660c15e390 0x7f660c0d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 -- 172.21.15.1:0/2365421433 >> v1:172.21.15.1:6789/0 conn(0x7f660c0b3320 legacy=0x7f660c0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f6610a1e640 1 -- 172.21.15.1:0/2365421433 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861622970 0 0) 0x7f6604001110 con 0x7f660c0b3320 2024-03-24T10:35:59.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.348+0000 7f661121f640 1 -- 172.21.15.1:0/2365421433 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c0d1e40 con 0x7f660c0d1000 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f661121f640 1 --2- 172.21.15.1:0/2365421433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c0d1000 0x7f660c0d29e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f660c0d3040 tx=0x7f65fc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f1ca66e078f2df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6610a1e640 1 -- 172.21.15.1:0/2365421433 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc031710 con 0x7f660c0d1000 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6610a1e640 1 -- 172.21.15.1:0/2365421433 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65fc02f050 con 0x7f660c0d1000 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6610a1e640 1 -- 172.21.15.1:0/2365421433 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc031a90 con 0x7f660c0d1000 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 -- 172.21.15.1:0/2365421433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c0d1000 msgr2=0x7f660c0d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 --2- 172.21.15.1:0/2365421433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c0d1000 0x7f660c0d29e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f660c0d3040 tx=0x7f65fc02b7b0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 -- 172.21.15.1:0/2365421433 shutdown_connections 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 --2- 172.21.15.1:0/2365421433 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c0d1000 0x7f660c0d29e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 --2- 172.21.15.1:0/2365421433 >> v2:172.21.15.3:3300/0 conn(0x7f660c15e390 0x7f660c0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.349+0000 7f6613c84640 1 -- 172.21.15.1:0/2365421433 >> 172.21.15.1:0/2365421433 conn(0x7f660c0d3ec0 msgr2=0x7f660c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.350+0000 7f6613c84640 1 -- 172.21.15.1:0/2365421433 shutdown_connections 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.350+0000 7f6613c84640 1 -- 172.21.15.1:0/2365421433 wait complete. 2024-03-24T10:35:59.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 Processor -- start 2024-03-24T10:35:59.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 -- start start 2024-03-24T10:35:59.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660c0b3320 0x7f660c0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660c0d1000 0x7f660c0a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0c9df0 con 0x7f660c15e390 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0b8dd0 con 0x7f660c0d1000 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.351+0000 7f6613c84640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0b90b0 con 0x7f660c0b3320 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38674/0 (socket says 172.21.15.1:38674) 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 -- 172.21.15.1:0/3072034827 learned_addr learned my addr 172.21.15.1:0/3072034827 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6611a20640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660c0b3320 0x7f660c0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f661121f640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660c0d1000 0x7f660c0a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 -- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660c0b3320 msgr2=0x7f660c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660c0b3320 0x7f660c0c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 -- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660c0d1000 msgr2=0x7f660c0a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660c0d1000 0x7f660c0a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 -- 172.21.15.1:0/3072034827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc009400 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6612221640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f660400c780 tx=0x7f660400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66040072d0 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6604007430 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c0b9390 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604001d10 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.352+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c0b9950 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.354+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f660c04f4e0 con 0x7f660c15e390 2024-03-24T10:35:59.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.354+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66040047e0 con 0x7f660c15e390 2024-03-24T10:35:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.354+0000 7f66027fc640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.354+0000 7f6611a20640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.354+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f6604089420 con 0x7f660c15e390 2024-03-24T10:35:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.355+0000 7f6611a20640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b23b0 tx=0x7f660800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.358+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6604053ed0 con 0x7f660c15e390 2024-03-24T10:35:59.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.635+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f660c0b4ba0 con 0x7f65f006b810 2024-03-24T10:35:59.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.643+0000 7f66027fc640 1 -- 172.21.15.1:0/3072034827 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251957 (secure 0 0 0) 0x7f6608007e40 con 0x7f65f006b810 2024-03-24T10:35:59.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:35:59.647 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 msgr2=0x7f65f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b23b0 tx=0x7f660800b040 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 msgr2=0x7f660c0a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f660400c780 tx=0x7f660400cc40 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 shutdown_connections 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f65f006b810 0x7f65f006dcd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f660c15e390 0x7f660c0a1850 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660c0d1000 0x7f660c0a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 --2- 172.21.15.1:0/3072034827 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f660c0b3320 0x7f660c0c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 >> 172.21.15.1:0/3072034827 conn(0x7f660c0d3ec0 msgr2=0x7f660c0d3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 shutdown_connections 2024-03-24T10:35:59.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.646+0000 7f6613c84640 1 -- 172.21.15.1:0/3072034827 wait complete. 2024-03-24T10:35:59.658 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-03-24T10:35:58.539958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238664,"kb_used_data":5864,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736056,"statfs":{"total":773094113280,"available":772849721344,"internally_reserved":0,"allocated":6004736,"data_stored":3161284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.925893"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631257+0000","last_change":"2024-03-24T10:35:51.636834+0000","last_active":"2024-03-24T10:35:52.631257+0000","last_peered":"2024-03-24T10:35:52.631257+0000","last_clean":"2024-03-24T10:35:52.631257+0000","last_became_active":"2024-03-24T10:35:51.636517+0000","last_became_peered":"2024-03-24T10:35:51.636517+0000","last_unstale":"2024-03-24T10:35:52.631257+0000","last_undegraded":"2024-03-24T10:35:52.631257+0000","last_fullsized":"2024-03-24T10:35:52.631257+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:35:47.733029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995477+0000","last_change":"2024-03-24T10:35:51.643583+0000","last_active":"2024-03-24T10:35:52.995477+0000","last_peered":"2024-03-24T10:35:52.995477+0000","last_clean":"2024-03-24T10:35:52.995477+0000","last_became_active":"2024-03-24T10:35:51.643271+0000","last_became_peered":"2024-03-24T10:35:51.643271+0000","last_unstale":"2024-03-24T10:35:52.995477+0000","last_undegraded":"2024-03-24T10:35:52.995477+0000","last_fullsized":"2024-03-24T10:35:52.995477+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:36.246303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993350+0000","last_change":"2024-03-24T10:35:51.645342+0000","last_active":"2024-03-24T10:35:52.993350+0000","last_peered":"2024-03-24T10:35:52.993350+0000","last_clean":"2024-03-24T10:35:52.993350+0000","last_became_active":"2024-03-24T10:35:51.645067+0000","last_became_peered":"2024-03-24T10:35:51.645067+0000","last_unstale":"2024-03-24T10:35:52.993350+0000","last_undegraded":"2024-03-24T10:35:52.993350+0000","last_fullsized":"2024-03-24T10:35:52.993350+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:58.611100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995194+0000","last_change":"2024-03-24T10:35:51.651873+0000","last_active":"2024-03-24T10:35:52.995194+0000","last_peered":"2024-03-24T10:35:52.995194+0000","last_clean":"2024-03-24T10:35:52.995194+0000","last_became_active":"2024-03-24T10:35:51.651601+0000","last_became_peered":"2024-03-24T10:35:51.651601+0000","last_unstale":"2024-03-24T10:35:52.995194+0000","last_undegraded":"2024-03-24T10:35:52.995194+0000","last_fullsized":"2024-03-24T10:35:52.995194+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:58.611100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994016+0000","last_change":"2024-03-24T10:35:51.625837+0000","last_active":"2024-03-24T10:35:52.994016+0000","last_peered":"2024-03-24T10:35:52.994016+0000","last_clean":"2024-03-24T10:35:52.994016+0000","last_became_active":"2024-03-24T10:35:51.625507+0000","last_became_peered":"2024-03-24T10:35:51.625507+0000","last_unstale":"2024-03-24T10:35:52.994016+0000","last_undegraded":"2024-03-24T10:35:52.994016+0000","last_fullsized":"2024-03-24T10:35:52.994016+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:46.063955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631203+0000","last_change":"2024-03-24T10:35:51.628068+0000","last_active":"2024-03-24T10:35:52.631203+0000","last_peered":"2024-03-24T10:35:52.631203+0000","last_clean":"2024-03-24T10:35:52.631203+0000","last_became_active":"2024-03-24T10:35:51.627416+0000","last_became_peered":"2024-03-24T10:35:51.627416+0000","last_unstale":"2024-03-24T10:35:52.631203+0000","last_undegraded":"2024-03-24T10:35:52.631203+0000","last_fullsized":"2024-03-24T10:35:52.631203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:37.527884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993005+0000","last_change":"2024-03-24T10:35:51.644106+0000","last_active":"2024-03-24T10:35:52.993005+0000","last_peered":"2024-03-24T10:35:52.993005+0000","last_clean":"2024-03-24T10:35:52.993005+0000","last_became_active":"2024-03-24T10:35:51.643776+0000","last_became_peered":"2024-03-24T10:35:51.643776+0000","last_unstale":"2024-03-24T10:35:52.993005+0000","last_undegraded":"2024-03-24T10:35:52.993005+0000","last_fullsized":"2024-03-24T10:35:52.993005+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:35:47.733029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633648+0000","last_change":"2024-03-24T10:35:51.651042+0000","last_active":"2024-03-24T10:35:52.633648+0000","last_peered":"2024-03-24T10:35:52.633648+0000","last_clean":"2024-03-24T10:35:52.633648+0000","last_became_active":"2024-03-24T10:35:51.650761+0000","last_became_peered":"2024-03-24T10:35:51.650761+0000","last_unstale":"2024-03-24T10:35:52.633648+0000","last_undegraded":"2024-03-24T10:35:52.633648+0000","last_fullsized":"2024-03-24T10:35:52.633648+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:08:03.316876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634089+0000","last_change":"2024-03-24T10:35:51.650789+0000","last_active":"2024-03-24T10:35:52.634089+0000","last_peered":"2024-03-24T10:35:52.634089+0000","last_clean":"2024-03-24T10:35:52.634089+0000","last_became_active":"2024-03-24T10:35:51.650443+0000","last_became_peered":"2024-03-24T10:35:51.650443+0000","last_unstale":"2024-03-24T10:35:52.634089+0000","last_undegraded":"2024-03-24T10:35:52.634089+0000","last_fullsized":"2024-03-24T10:35:52.634089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:34.805082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631169+0000","last_change":"2024-03-24T10:35:51.628718+0000","last_active":"2024-03-24T10:35:52.631169+0000","last_peered":"2024-03-24T10:35:52.631169+0000","last_clean":"2024-03-24T10:35:52.631169+0000","last_became_active":"2024-03-24T10:35:51.628456+0000","last_became_peered":"2024-03-24T10:35:51.628456+0000","last_unstale":"2024-03-24T10:35:52.631169+0000","last_undegraded":"2024-03-24T10:35:52.631169+0000","last_fullsized":"2024-03-24T10:35:52.631169+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:02.215491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634727+0000","last_change":"2024-03-24T10:35:51.641161+0000","last_active":"2024-03-24T10:35:52.634727+0000","last_peered":"2024-03-24T10:35:52.634727+0000","last_clean":"2024-03-24T10:35:52.634727+0000","last_became_active":"2024-03-24T10:35:51.640882+0000","last_became_peered":"2024-03-24T10:35:51.640882+0000","last_unstale":"2024-03-24T10:35:52.634727+0000","last_undegraded":"2024-03-24T10:35:52.634727+0000","last_fullsized":"2024-03-24T10:35:52.634727+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:29:26.313796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633917+0000","last_change":"2024-03-24T10:35:51.649815+0000","last_active":"2024-03-24T10:35:52.633917+0000","last_peered":"2024-03-24T10:35:52.633917+0000","last_clean":"2024-03-24T10:35:52.633917+0000","last_became_active":"2024-03-24T10:35:51.649523+0000","last_became_peered":"2024-03-24T10:35:51.649523+0000","last_unstale":"2024-03-24T10:35:52.633917+0000","last_undegraded":"2024-03-24T10:35:52.633917+0000","last_fullsized":"2024-03-24T10:35:52.633917+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.243807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631386+0000","last_change":"2024-03-24T10:35:51.625541+0000","last_active":"2024-03-24T10:35:52.631386+0000","last_peered":"2024-03-24T10:35:52.631386+0000","last_clean":"2024-03-24T10:35:52.631386+0000","last_became_active":"2024-03-24T10:35:51.625162+0000","last_became_peered":"2024-03-24T10:35:51.625162+0000","last_unstale":"2024-03-24T10:35:52.631386+0000","last_undegraded":"2024-03-24T10:35:52.631386+0000","last_fullsized":"2024-03-24T10:35:52.631386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:02.899497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993152+0000","last_change":"2024-03-24T10:35:51.641707+0000","last_active":"2024-03-24T10:35:52.993152+0000","last_peered":"2024-03-24T10:35:52.993152+0000","last_clean":"2024-03-24T10:35:52.993152+0000","last_became_active":"2024-03-24T10:35:51.641388+0000","last_became_peered":"2024-03-24T10:35:51.641388+0000","last_unstale":"2024-03-24T10:35:52.993152+0000","last_undegraded":"2024-03-24T10:35:52.993152+0000","last_fullsized":"2024-03-24T10:35:52.993152+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:47.095893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994438+0000","last_change":"2024-03-24T10:35:51.635924+0000","last_active":"2024-03-24T10:35:52.994438+0000","last_peered":"2024-03-24T10:35:52.994438+0000","last_clean":"2024-03-24T10:35:52.994438+0000","last_became_active":"2024-03-24T10:35:51.634885+0000","last_became_peered":"2024-03-24T10:35:51.634885+0000","last_unstale":"2024-03-24T10:35:52.994438+0000","last_undegraded":"2024-03-24T10:35:52.994438+0000","last_fullsized":"2024-03-24T10:35:52.994438+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:02.899497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631023+0000","last_change":"2024-03-24T10:35:51.624655+0000","last_active":"2024-03-24T10:35:52.631023+0000","last_peered":"2024-03-24T10:35:52.631023+0000","last_clean":"2024-03-24T10:35:52.631023+0000","last_became_active":"2024-03-24T10:35:51.624046+0000","last_became_peered":"2024-03-24T10:35:51.624046+0000","last_unstale":"2024-03-24T10:35:52.631023+0000","last_undegraded":"2024-03-24T10:35:52.631023+0000","last_fullsized":"2024-03-24T10:35:52.631023+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:41.367683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993661+0000","last_change":"2024-03-24T10:35:51.634085+0000","last_active":"2024-03-24T10:35:52.993661+0000","last_peered":"2024-03-24T10:35:52.993661+0000","last_clean":"2024-03-24T10:35:52.993661+0000","last_became_active":"2024-03-24T10:35:51.633781+0000","last_became_peered":"2024-03-24T10:35:51.633781+0000","last_unstale":"2024-03-24T10:35:52.993661+0000","last_undegraded":"2024-03-24T10:35:52.993661+0000","last_fullsized":"2024-03-24T10:35:52.993661+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:33:26.931679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995565+0000","last_change":"2024-03-24T10:35:51.641005+0000","last_active":"2024-03-24T10:35:52.995565+0000","last_peered":"2024-03-24T10:35:52.995565+0000","last_clean":"2024-03-24T10:35:52.995565+0000","last_became_active":"2024-03-24T10:35:51.640702+0000","last_became_peered":"2024-03-24T10:35:51.640702+0000","last_unstale":"2024-03-24T10:35:52.995565+0000","last_undegraded":"2024-03-24T10:35:52.995565+0000","last_fullsized":"2024-03-24T10:35:52.995565+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:06.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993419+0000","last_change":"2024-03-24T10:35:51.629988+0000","last_active":"2024-03-24T10:35:52.993419+0000","last_peered":"2024-03-24T10:35:52.993419+0000","last_clean":"2024-03-24T10:35:52.993419+0000","last_became_active":"2024-03-24T10:35:51.629657+0000","last_became_peered":"2024-03-24T10:35:51.629657+0000","last_unstale":"2024-03-24T10:35:52.993419+0000","last_undegraded":"2024-03-24T10:35:52.993419+0000","last_fullsized":"2024-03-24T10:35:52.993419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:24:12.690643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994531+0000","last_change":"2024-03-24T10:35:51.627579+0000","last_active":"2024-03-24T10:35:52.994531+0000","last_peered":"2024-03-24T10:35:52.994531+0000","last_clean":"2024-03-24T10:35:52.994531+0000","last_became_active":"2024-03-24T10:35:51.627308+0000","last_became_peered":"2024-03-24T10:35:51.627308+0000","last_unstale":"2024-03-24T10:35:52.994531+0000","last_undegraded":"2024-03-24T10:35:52.994531+0000","last_fullsized":"2024-03-24T10:35:52.994531+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:27.829972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994650+0000","last_change":"2024-03-24T10:35:51.626124+0000","last_active":"2024-03-24T10:35:52.994650+0000","last_peered":"2024-03-24T10:35:52.994650+0000","last_clean":"2024-03-24T10:35:52.994650+0000","last_became_active":"2024-03-24T10:35:51.625806+0000","last_became_peered":"2024-03-24T10:35:51.625806+0000","last_unstale":"2024-03-24T10:35:52.994650+0000","last_undegraded":"2024-03-24T10:35:52.994650+0000","last_fullsized":"2024-03-24T10:35:52.994650+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:58.454041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633545+0000","last_change":"2024-03-24T10:35:51.651097+0000","last_active":"2024-03-24T10:35:52.633545+0000","last_peered":"2024-03-24T10:35:52.633545+0000","last_clean":"2024-03-24T10:35:52.633545+0000","last_became_active":"2024-03-24T10:35:51.650822+0000","last_became_peered":"2024-03-24T10:35:51.650822+0000","last_unstale":"2024-03-24T10:35:52.633545+0000","last_undegraded":"2024-03-24T10:35:52.633545+0000","last_fullsized":"2024-03-24T10:35:52.633545+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:35:56.758472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634464+0000","last_change":"2024-03-24T10:35:51.637020+0000","last_active":"2024-03-24T10:35:52.634464+0000","last_peered":"2024-03-24T10:35:52.634464+0000","last_clean":"2024-03-24T10:35:52.634464+0000","last_became_active":"2024-03-24T10:35:51.636715+0000","last_became_peered":"2024-03-24T10:35:51.636715+0000","last_unstale":"2024-03-24T10:35:52.634464+0000","last_undegraded":"2024-03-24T10:35:52.634464+0000","last_fullsized":"2024-03-24T10:35:52.634464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:16.559012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.992930+0000","last_change":"2024-03-24T10:35:51.630493+0000","last_active":"2024-03-24T10:35:52.992930+0000","last_peered":"2024-03-24T10:35:52.992930+0000","last_clean":"2024-03-24T10:35:52.992930+0000","last_became_active":"2024-03-24T10:35:51.630219+0000","last_became_peered":"2024-03-24T10:35:51.630219+0000","last_unstale":"2024-03-24T10:35:52.992930+0000","last_undegraded":"2024-03-24T10:35:52.992930+0000","last_fullsized":"2024-03-24T10:35:52.992930+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:39.341896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993676+0000","last_change":"2024-03-24T10:35:51.644138+0000","last_active":"2024-03-24T10:35:52.993676+0000","last_peered":"2024-03-24T10:35:52.993676+0000","last_clean":"2024-03-24T10:35:52.993676+0000","last_became_active":"2024-03-24T10:35:51.643865+0000","last_became_peered":"2024-03-24T10:35:51.643865+0000","last_unstale":"2024-03-24T10:35:52.993676+0000","last_undegraded":"2024-03-24T10:35:52.993676+0000","last_fullsized":"2024-03-24T10:35:52.993676+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:29.552537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633725+0000","last_change":"2024-03-24T10:35:51.643480+0000","last_active":"2024-03-24T10:35:52.633725+0000","last_peered":"2024-03-24T10:35:52.633725+0000","last_clean":"2024-03-24T10:35:52.633725+0000","last_became_active":"2024-03-24T10:35:51.639656+0000","last_became_peered":"2024-03-24T10:35:51.639656+0000","last_unstale":"2024-03-24T10:35:52.633725+0000","last_undegraded":"2024-03-24T10:35:52.633725+0000","last_fullsized":"2024-03-24T10:35:52.633725+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:39:17.418821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631566+0000","last_change":"2024-03-24T10:35:51.636160+0000","last_active":"2024-03-24T10:35:52.631566+0000","last_peered":"2024-03-24T10:35:52.631566+0000","last_clean":"2024-03-24T10:35:52.631566+0000","last_became_active":"2024-03-24T10:35:51.635870+0000","last_became_peered":"2024-03-24T10:35:51.635870+0000","last_unstale":"2024-03-24T10:35:52.631566+0000","last_undegraded":"2024-03-24T10:35:52.631566+0000","last_fullsized":"2024-03-24T10:35:52.631566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:47.095893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994689+0000","last_change":"2024-03-24T10:35:51.632890+0000","last_active":"2024-03-24T10:35:52.994689+0000","last_peered":"2024-03-24T10:35:52.994689+0000","last_clean":"2024-03-24T10:35:52.994689+0000","last_became_active":"2024-03-24T10:35:51.632617+0000","last_became_peered":"2024-03-24T10:35:51.632617+0000","last_unstale":"2024-03-24T10:35:52.994689+0000","last_undegraded":"2024-03-24T10:35:52.994689+0000","last_fullsized":"2024-03-24T10:35:52.994689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:46.063955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993117+0000","last_change":"2024-03-24T10:35:51.631980+0000","last_active":"2024-03-24T10:35:52.993117+0000","last_peered":"2024-03-24T10:35:52.993117+0000","last_clean":"2024-03-24T10:35:52.993117+0000","last_became_active":"2024-03-24T10:35:51.631636+0000","last_became_peered":"2024-03-24T10:35:51.631636+0000","last_unstale":"2024-03-24T10:35:52.993117+0000","last_undegraded":"2024-03-24T10:35:52.993117+0000","last_fullsized":"2024-03-24T10:35:52.993117+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:02.899497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994885+0000","last_change":"2024-03-24T10:35:51.641141+0000","last_active":"2024-03-24T10:35:52.994885+0000","last_peered":"2024-03-24T10:35:52.994885+0000","last_clean":"2024-03-24T10:35:52.994885+0000","last_became_active":"2024-03-24T10:35:51.640867+0000","last_became_peered":"2024-03-24T10:35:51.640867+0000","last_unstale":"2024-03-24T10:35:52.994885+0000","last_undegraded":"2024-03-24T10:35:52.994885+0000","last_fullsized":"2024-03-24T10:35:52.994885+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.469020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993077+0000","last_change":"2024-03-24T10:35:51.642760+0000","last_active":"2024-03-24T10:35:52.993077+0000","last_peered":"2024-03-24T10:35:52.993077+0000","last_clean":"2024-03-24T10:35:52.993077+0000","last_became_active":"2024-03-24T10:35:51.642427+0000","last_became_peered":"2024-03-24T10:35:51.642427+0000","last_unstale":"2024-03-24T10:35:52.993077+0000","last_undegraded":"2024-03-24T10:35:52.993077+0000","last_fullsized":"2024-03-24T10:35:52.993077+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:27:04.075934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994179+0000","last_change":"2024-03-24T10:35:51.647733+0000","last_active":"2024-03-24T10:35:52.994179+0000","last_peered":"2024-03-24T10:35:52.994179+0000","last_clean":"2024-03-24T10:35:52.994179+0000","last_became_active":"2024-03-24T10:35:51.647448+0000","last_became_peered":"2024-03-24T10:35:51.647448+0000","last_unstale":"2024-03-24T10:35:52.994179+0000","last_undegraded":"2024-03-24T10:35:52.994179+0000","last_fullsized":"2024-03-24T10:35:52.994179+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:12:41.367683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994862+0000","last_change":"2024-03-24T10:35:51.636007+0000","last_active":"2024-03-24T10:35:52.994862+0000","last_peered":"2024-03-24T10:35:52.994862+0000","last_clean":"2024-03-24T10:35:52.994862+0000","last_became_active":"2024-03-24T10:35:51.635683+0000","last_became_peered":"2024-03-24T10:35:51.635683+0000","last_unstale":"2024-03-24T10:35:52.994862+0000","last_undegraded":"2024-03-24T10:35:52.994862+0000","last_fullsized":"2024-03-24T10:35:52.994862+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:47.095893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994127+0000","last_change":"2024-03-24T10:35:50.635274+0000","last_active":"2024-03-24T10:35:52.994127+0000","last_peered":"2024-03-24T10:35:52.994127+0000","last_clean":"2024-03-24T10:35:52.994127+0000","last_became_active":"2024-03-24T10:35:50.634934+0000","last_became_peered":"2024-03-24T10:35:50.634934+0000","last_unstale":"2024-03-24T10:35:52.994127+0000","last_undegraded":"2024-03-24T10:35:52.994127+0000","last_fullsized":"2024-03-24T10:35:52.994127+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:07:48.903201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994572+0000","last_change":"2024-03-24T10:35:51.642702+0000","last_active":"2024-03-24T10:35:52.994572+0000","last_peered":"2024-03-24T10:35:52.994572+0000","last_clean":"2024-03-24T10:35:52.994572+0000","last_became_active":"2024-03-24T10:35:51.642433+0000","last_became_peered":"2024-03-24T10:35:51.642433+0000","last_unstale":"2024-03-24T10:35:52.994572+0000","last_undegraded":"2024-03-24T10:35:52.994572+0000","last_fullsized":"2024-03-24T10:35:52.994572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:37.527884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631167+0000","last_change":"2024-03-24T10:35:50.633891+0000","last_active":"2024-03-24T10:35:52.631167+0000","last_peered":"2024-03-24T10:35:52.631167+0000","last_clean":"2024-03-24T10:35:52.631167+0000","last_became_active":"2024-03-24T10:35:50.633604+0000","last_became_peered":"2024-03-24T10:35:50.633604+0000","last_unstale":"2024-03-24T10:35:52.631167+0000","last_undegraded":"2024-03-24T10:35:52.631167+0000","last_fullsized":"2024-03-24T10:35:52.631167+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:10.078517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994551+0000","last_change":"2024-03-24T10:35:51.638888+0000","last_active":"2024-03-24T10:35:52.994551+0000","last_peered":"2024-03-24T10:35:52.994551+0000","last_clean":"2024-03-24T10:35:52.994551+0000","last_became_active":"2024-03-24T10:35:51.638602+0000","last_became_peered":"2024-03-24T10:35:51.638602+0000","last_unstale":"2024-03-24T10:35:52.994551+0000","last_undegraded":"2024-03-24T10:35:52.994551+0000","last_fullsized":"2024-03-24T10:35:52.994551+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:21:00.512786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631518+0000","last_change":"2024-03-24T10:35:50.637492+0000","last_active":"2024-03-24T10:35:52.631518+0000","last_peered":"2024-03-24T10:35:52.631518+0000","last_clean":"2024-03-24T10:35:52.631518+0000","last_became_active":"2024-03-24T10:35:50.637225+0000","last_became_peered":"2024-03-24T10:35:50.637225+0000","last_unstale":"2024-03-24T10:35:52.631518+0000","last_undegraded":"2024-03-24T10:35:52.631518+0000","last_fullsized":"2024-03-24T10:35:52.631518+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:47.712468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633584+0000","last_change":"2024-03-24T10:35:51.642726+0000","last_active":"2024-03-24T10:35:52.633584+0000","last_peered":"2024-03-24T10:35:52.633584+0000","last_clean":"2024-03-24T10:35:52.633584+0000","last_became_active":"2024-03-24T10:35:51.639644+0000","last_became_peered":"2024-03-24T10:35:51.639644+0000","last_unstale":"2024-03-24T10:35:52.633584+0000","last_undegraded":"2024-03-24T10:35:52.633584+0000","last_fullsized":"2024-03-24T10:35:52.633584+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:03.836208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994643+0000","last_change":"2024-03-24T10:35:50.632990+0000","last_active":"2024-03-24T10:35:52.994643+0000","last_peered":"2024-03-24T10:35:52.994643+0000","last_clean":"2024-03-24T10:35:52.994643+0000","last_became_active":"2024-03-24T10:35:50.632635+0000","last_became_peered":"2024-03-24T10:35:50.632635+0000","last_unstale":"2024-03-24T10:35:52.994643+0000","last_undegraded":"2024-03-24T10:35:52.994643+0000","last_fullsized":"2024-03-24T10:35:52.994643+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:10.263999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993916+0000","last_change":"2024-03-24T10:35:51.631142+0000","last_active":"2024-03-24T10:35:52.993916+0000","last_peered":"2024-03-24T10:35:52.993916+0000","last_clean":"2024-03-24T10:35:52.993916+0000","last_became_active":"2024-03-24T10:35:51.630309+0000","last_became_peered":"2024-03-24T10:35:51.630309+0000","last_unstale":"2024-03-24T10:35:52.993916+0000","last_undegraded":"2024-03-24T10:35:52.993916+0000","last_fullsized":"2024-03-24T10:35:52.993916+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:58.454041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993084+0000","last_change":"2024-03-24T10:35:50.638378+0000","last_active":"2024-03-24T10:35:52.993084+0000","last_peered":"2024-03-24T10:35:52.993084+0000","last_clean":"2024-03-24T10:35:52.993084+0000","last_became_active":"2024-03-24T10:35:50.638077+0000","last_became_peered":"2024-03-24T10:35:50.638077+0000","last_unstale":"2024-03-24T10:35:52.993084+0000","last_undegraded":"2024-03-24T10:35:52.993084+0000","last_fullsized":"2024-03-24T10:35:52.993084+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:57.449362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995180+0000","last_change":"2024-03-24T10:35:51.641271+0000","last_active":"2024-03-24T10:35:52.995180+0000","last_peered":"2024-03-24T10:35:52.995180+0000","last_clean":"2024-03-24T10:35:52.995180+0000","last_became_active":"2024-03-24T10:35:51.640945+0000","last_became_peered":"2024-03-24T10:35:51.640945+0000","last_unstale":"2024-03-24T10:35:52.995180+0000","last_undegraded":"2024-03-24T10:35:52.995180+0000","last_fullsized":"2024-03-24T10:35:52.995180+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:26.341866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993922+0000","last_change":"2024-03-24T10:35:50.636470+0000","last_active":"2024-03-24T10:35:52.993922+0000","last_peered":"2024-03-24T10:35:52.993922+0000","last_clean":"2024-03-24T10:35:52.993922+0000","last_became_active":"2024-03-24T10:35:50.636139+0000","last_became_peered":"2024-03-24T10:35:50.636139+0000","last_unstale":"2024-03-24T10:35:52.993922+0000","last_undegraded":"2024-03-24T10:35:52.993922+0000","last_fullsized":"2024-03-24T10:35:52.993922+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:16.151719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633974+0000","last_change":"2024-03-24T10:35:51.644390+0000","last_active":"2024-03-24T10:35:52.633974+0000","last_peered":"2024-03-24T10:35:52.633974+0000","last_clean":"2024-03-24T10:35:52.633974+0000","last_became_active":"2024-03-24T10:35:51.644115+0000","last_became_peered":"2024-03-24T10:35:51.644115+0000","last_unstale":"2024-03-24T10:35:52.633974+0000","last_undegraded":"2024-03-24T10:35:52.633974+0000","last_fullsized":"2024-03-24T10:35:52.633974+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:59:12.895218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993592+0000","last_change":"2024-03-24T10:35:50.635522+0000","last_active":"2024-03-24T10:35:52.993592+0000","last_peered":"2024-03-24T10:35:52.993592+0000","last_clean":"2024-03-24T10:35:52.993592+0000","last_became_active":"2024-03-24T10:35:50.635036+0000","last_became_peered":"2024-03-24T10:35:50.635036+0000","last_unstale":"2024-03-24T10:35:52.993592+0000","last_undegraded":"2024-03-24T10:35:52.993592+0000","last_fullsized":"2024-03-24T10:35:52.993592+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:46:18.495739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995259+0000","last_change":"2024-03-24T10:35:51.646621+0000","last_active":"2024-03-24T10:35:52.995259+0000","last_peered":"2024-03-24T10:35:52.995259+0000","last_clean":"2024-03-24T10:35:52.995259+0000","last_became_active":"2024-03-24T10:35:51.646317+0000","last_became_peered":"2024-03-24T10:35:51.646317+0000","last_unstale":"2024-03-24T10:35:52.995259+0000","last_undegraded":"2024-03-24T10:35:52.995259+0000","last_fullsized":"2024-03-24T10:35:52.995259+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:24:12.690643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.992969+0000","last_change":"2024-03-24T10:35:50.636110+0000","last_active":"2024-03-24T10:35:52.992969+0000","last_peered":"2024-03-24T10:35:52.992969+0000","last_clean":"2024-03-24T10:35:52.992969+0000","last_became_active":"2024-03-24T10:35:50.635804+0000","last_became_peered":"2024-03-24T10:35:50.635804+0000","last_unstale":"2024-03-24T10:35:52.992969+0000","last_undegraded":"2024-03-24T10:35:52.992969+0000","last_fullsized":"2024-03-24T10:35:52.992969+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:01.118876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993660+0000","last_change":"2024-03-24T10:35:51.646189+0000","last_active":"2024-03-24T10:35:52.993660+0000","last_peered":"2024-03-24T10:35:52.993660+0000","last_clean":"2024-03-24T10:35:52.993660+0000","last_became_active":"2024-03-24T10:35:51.645809+0000","last_became_peered":"2024-03-24T10:35:51.645809+0000","last_unstale":"2024-03-24T10:35:52.993660+0000","last_undegraded":"2024-03-24T10:35:52.993660+0000","last_fullsized":"2024-03-24T10:35:52.993660+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:27.829972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995390+0000","last_change":"2024-03-24T10:35:50.630099+0000","last_active":"2024-03-24T10:35:52.995390+0000","last_peered":"2024-03-24T10:35:52.995390+0000","last_clean":"2024-03-24T10:35:52.995390+0000","last_became_active":"2024-03-24T10:35:50.629665+0000","last_became_peered":"2024-03-24T10:35:50.629665+0000","last_unstale":"2024-03-24T10:35:52.995390+0000","last_undegraded":"2024-03-24T10:35:52.995390+0000","last_fullsized":"2024-03-24T10:35:52.995390+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:55.782184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994198+0000","last_change":"2024-03-24T10:35:51.638773+0000","last_active":"2024-03-24T10:35:52.994198+0000","last_peered":"2024-03-24T10:35:52.994198+0000","last_clean":"2024-03-24T10:35:52.994198+0000","last_became_active":"2024-03-24T10:35:51.637452+0000","last_became_peered":"2024-03-24T10:35:51.637452+0000","last_unstale":"2024-03-24T10:35:52.994198+0000","last_undegraded":"2024-03-24T10:35:52.994198+0000","last_fullsized":"2024-03-24T10:35:52.994198+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:45.637553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994816+0000","last_change":"2024-03-24T10:35:50.638305+0000","last_active":"2024-03-24T10:35:52.994816+0000","last_peered":"2024-03-24T10:35:52.994816+0000","last_clean":"2024-03-24T10:35:52.994816+0000","last_became_active":"2024-03-24T10:35:50.637768+0000","last_became_peered":"2024-03-24T10:35:50.637768+0000","last_unstale":"2024-03-24T10:35:52.994816+0000","last_undegraded":"2024-03-24T10:35:52.994816+0000","last_fullsized":"2024-03-24T10:35:52.994816+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:26.730054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993186+0000","last_change":"2024-03-24T10:35:51.631241+0000","last_active":"2024-03-24T10:35:52.993186+0000","last_peered":"2024-03-24T10:35:52.993186+0000","last_clean":"2024-03-24T10:35:52.993186+0000","last_became_active":"2024-03-24T10:35:51.630987+0000","last_became_peered":"2024-03-24T10:35:51.630987+0000","last_unstale":"2024-03-24T10:35:52.993186+0000","last_undegraded":"2024-03-24T10:35:52.993186+0000","last_fullsized":"2024-03-24T10:35:52.993186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.469020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":17,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995314+0000","last_change":"2024-03-24T10:35:50.631873+0000","last_active":"2024-03-24T10:35:52.995314+0000","last_peered":"2024-03-24T10:35:52.995314+0000","last_clean":"2024-03-24T10:35:52.995314+0000","last_became_active":"2024-03-24T10:35:50.631567+0000","last_became_peered":"2024-03-24T10:35:50.631567+0000","last_unstale":"2024-03-24T10:35:52.995314+0000","last_undegraded":"2024-03-24T10:35:52.995314+0000","last_fullsized":"2024-03-24T10:35:52.995314+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:40:41.063471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994116+0000","last_change":"2024-03-24T10:35:51.633638+0000","last_active":"2024-03-24T10:35:52.994116+0000","last_peered":"2024-03-24T10:35:52.994116+0000","last_clean":"2024-03-24T10:35:52.994116+0000","last_became_active":"2024-03-24T10:35:51.633360+0000","last_became_peered":"2024-03-24T10:35:51.633360+0000","last_unstale":"2024-03-24T10:35:52.994116+0000","last_undegraded":"2024-03-24T10:35:52.994116+0000","last_fullsized":"2024-03-24T10:35:52.994116+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:13.011680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633925+0000","last_change":"2024-03-24T10:35:50.626864+0000","last_active":"2024-03-24T10:35:52.633925+0000","last_peered":"2024-03-24T10:35:52.633925+0000","last_clean":"2024-03-24T10:35:52.633925+0000","last_became_active":"2024-03-24T10:35:50.626556+0000","last_became_peered":"2024-03-24T10:35:50.626556+0000","last_unstale":"2024-03-24T10:35:52.633925+0000","last_undegraded":"2024-03-24T10:35:52.633925+0000","last_fullsized":"2024-03-24T10:35:52.633925+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:36:01.894817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994395+0000","last_change":"2024-03-24T10:35:51.640008+0000","last_active":"2024-03-24T10:35:52.994395+0000","last_peered":"2024-03-24T10:35:52.994395+0000","last_clean":"2024-03-24T10:35:52.994395+0000","last_became_active":"2024-03-24T10:35:51.639677+0000","last_became_peered":"2024-03-24T10:35:51.639677+0000","last_unstale":"2024-03-24T10:35:52.994395+0000","last_undegraded":"2024-03-24T10:35:52.994395+0000","last_fullsized":"2024-03-24T10:35:52.994395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:02.215491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993215+0000","last_change":"2024-03-24T10:35:50.637003+0000","last_active":"2024-03-24T10:35:52.993215+0000","last_peered":"2024-03-24T10:35:52.993215+0000","last_clean":"2024-03-24T10:35:52.993215+0000","last_became_active":"2024-03-24T10:35:50.636716+0000","last_became_peered":"2024-03-24T10:35:50.636716+0000","last_unstale":"2024-03-24T10:35:52.993215+0000","last_undegraded":"2024-03-24T10:35:52.993215+0000","last_fullsized":"2024-03-24T10:35:52.993215+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:08:45.059275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993507+0000","last_change":"2024-03-24T10:35:51.633157+0000","last_active":"2024-03-24T10:35:52.993507+0000","last_peered":"2024-03-24T10:35:52.993507+0000","last_clean":"2024-03-24T10:35:52.993507+0000","last_became_active":"2024-03-24T10:35:51.632871+0000","last_became_peered":"2024-03-24T10:35:51.632871+0000","last_unstale":"2024-03-24T10:35:52.993507+0000","last_undegraded":"2024-03-24T10:35:52.993507+0000","last_fullsized":"2024-03-24T10:35:52.993507+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:57:08.146249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994666+0000","last_change":"2024-03-24T10:35:50.637095+0000","last_active":"2024-03-24T10:35:52.994666+0000","last_peered":"2024-03-24T10:35:52.994666+0000","last_clean":"2024-03-24T10:35:52.994666+0000","last_became_active":"2024-03-24T10:35:50.636717+0000","last_became_peered":"2024-03-24T10:35:50.636717+0000","last_unstale":"2024-03-24T10:35:52.994666+0000","last_undegraded":"2024-03-24T10:35:52.994666+0000","last_fullsized":"2024-03-24T10:35:52.994666+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:27:14.345670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634181+0000","last_change":"2024-03-24T10:35:51.646327+0000","last_active":"2024-03-24T10:35:52.634181+0000","last_peered":"2024-03-24T10:35:52.634181+0000","last_clean":"2024-03-24T10:35:52.634181+0000","last_became_active":"2024-03-24T10:35:51.646065+0000","last_became_peered":"2024-03-24T10:35:51.646065+0000","last_unstale":"2024-03-24T10:35:52.634181+0000","last_undegraded":"2024-03-24T10:35:52.634181+0000","last_fullsized":"2024-03-24T10:35:52.634181+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:38.531011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994535+0000","last_change":"2024-03-24T10:35:50.637013+0000","last_active":"2024-03-24T10:35:52.994535+0000","last_peered":"2024-03-24T10:35:52.994535+0000","last_clean":"2024-03-24T10:35:52.994535+0000","last_became_active":"2024-03-24T10:35:50.636281+0000","last_became_peered":"2024-03-24T10:35:50.636281+0000","last_unstale":"2024-03-24T10:35:52.994535+0000","last_undegraded":"2024-03-24T10:35:52.994535+0000","last_fullsized":"2024-03-24T10:35:52.994535+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:23:26.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993537+0000","last_change":"2024-03-24T10:35:51.641571+0000","last_active":"2024-03-24T10:35:52.993537+0000","last_peered":"2024-03-24T10:35:52.993537+0000","last_clean":"2024-03-24T10:35:52.993537+0000","last_became_active":"2024-03-24T10:35:51.641262+0000","last_became_peered":"2024-03-24T10:35:51.641262+0000","last_unstale":"2024-03-24T10:35:52.993537+0000","last_undegraded":"2024-03-24T10:35:52.993537+0000","last_fullsized":"2024-03-24T10:35:52.993537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:58.649674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":22,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633902+0000","last_change":"2024-03-24T10:35:50.626837+0000","last_active":"2024-03-24T10:35:52.633902+0000","last_peered":"2024-03-24T10:35:52.633902+0000","last_clean":"2024-03-24T10:35:52.633902+0000","last_became_active":"2024-03-24T10:35:50.626538+0000","last_became_peered":"2024-03-24T10:35:50.626538+0000","last_unstale":"2024-03-24T10:35:52.633902+0000","last_undegraded":"2024-03-24T10:35:52.633902+0000","last_fullsized":"2024-03-24T10:35:52.633902+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:36.523204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633806+0000","last_change":"2024-03-24T10:35:51.647698+0000","last_active":"2024-03-24T10:35:52.633806+0000","last_peered":"2024-03-24T10:35:52.633806+0000","last_clean":"2024-03-24T10:35:52.633806+0000","last_became_active":"2024-03-24T10:35:51.647421+0000","last_became_peered":"2024-03-24T10:35:51.647421+0000","last_unstale":"2024-03-24T10:35:52.633806+0000","last_undegraded":"2024-03-24T10:35:52.633806+0000","last_fullsized":"2024-03-24T10:35:52.633806+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:43.345280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993745+0000","last_change":"2024-03-24T10:35:50.638452+0000","last_active":"2024-03-24T10:35:52.993745+0000","last_peered":"2024-03-24T10:35:52.993745+0000","last_clean":"2024-03-24T10:35:52.993745+0000","last_became_active":"2024-03-24T10:35:50.636353+0000","last_became_peered":"2024-03-24T10:35:50.636353+0000","last_unstale":"2024-03-24T10:35:52.993745+0000","last_undegraded":"2024-03-24T10:35:52.993745+0000","last_fullsized":"2024-03-24T10:35:52.993745+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:53:10.078517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631101+0000","last_change":"2024-03-24T10:35:51.629210+0000","last_active":"2024-03-24T10:35:52.631101+0000","last_peered":"2024-03-24T10:35:52.631101+0000","last_clean":"2024-03-24T10:35:52.631101+0000","last_became_active":"2024-03-24T10:35:51.628932+0000","last_became_peered":"2024-03-24T10:35:51.628932+0000","last_unstale":"2024-03-24T10:35:52.631101+0000","last_undegraded":"2024-03-24T10:35:52.631101+0000","last_fullsized":"2024-03-24T10:35:52.631101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:45.637553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634750+0000","last_change":"2024-03-24T10:35:50.628002+0000","last_active":"2024-03-24T10:35:52.634750+0000","last_peered":"2024-03-24T10:35:52.634750+0000","last_clean":"2024-03-24T10:35:52.634750+0000","last_became_active":"2024-03-24T10:35:50.627615+0000","last_became_peered":"2024-03-24T10:35:50.627615+0000","last_unstale":"2024-03-24T10:35:52.634750+0000","last_undegraded":"2024-03-24T10:35:52.634750+0000","last_fullsized":"2024-03-24T10:35:52.634750+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:07.837632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993822+0000","last_change":"2024-03-24T10:35:51.630872+0000","last_active":"2024-03-24T10:35:52.993822+0000","last_peered":"2024-03-24T10:35:52.993822+0000","last_clean":"2024-03-24T10:35:52.993822+0000","last_became_active":"2024-03-24T10:35:51.630566+0000","last_became_peered":"2024-03-24T10:35:51.630566+0000","last_unstale":"2024-03-24T10:35:52.993822+0000","last_undegraded":"2024-03-24T10:35:52.993822+0000","last_fullsized":"2024-03-24T10:35:52.993822+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:21:00.512786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994506+0000","last_change":"2024-03-24T10:35:50.632552+0000","last_active":"2024-03-24T10:35:52.994506+0000","last_peered":"2024-03-24T10:35:52.994506+0000","last_clean":"2024-03-24T10:35:52.994506+0000","last_became_active":"2024-03-24T10:35:50.632243+0000","last_became_peered":"2024-03-24T10:35:50.632243+0000","last_unstale":"2024-03-24T10:35:52.994506+0000","last_undegraded":"2024-03-24T10:35:52.994506+0000","last_fullsized":"2024-03-24T10:35:52.994506+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:33:50.611475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631182+0000","last_change":"2024-03-24T10:35:51.633341+0000","last_active":"2024-03-24T10:35:52.631182+0000","last_peered":"2024-03-24T10:35:52.631182+0000","last_clean":"2024-03-24T10:35:52.631182+0000","last_became_active":"2024-03-24T10:35:51.632971+0000","last_became_peered":"2024-03-24T10:35:51.632971+0000","last_unstale":"2024-03-24T10:35:52.631182+0000","last_undegraded":"2024-03-24T10:35:52.631182+0000","last_fullsized":"2024-03-24T10:35:52.631182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:42.375488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994791+0000","last_change":"2024-03-24T10:35:50.632996+0000","last_active":"2024-03-24T10:35:52.994791+0000","last_peered":"2024-03-24T10:35:52.994791+0000","last_clean":"2024-03-24T10:35:52.994791+0000","last_became_active":"2024-03-24T10:35:50.632669+0000","last_became_peered":"2024-03-24T10:35:50.632669+0000","last_unstale":"2024-03-24T10:35:52.994791+0000","last_undegraded":"2024-03-24T10:35:52.994791+0000","last_fullsized":"2024-03-24T10:35:52.994791+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:41.493981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994549+0000","last_change":"2024-03-24T10:35:51.638159+0000","last_active":"2024-03-24T10:35:52.994549+0000","last_peered":"2024-03-24T10:35:52.994549+0000","last_clean":"2024-03-24T10:35:52.994549+0000","last_became_active":"2024-03-24T10:35:51.637846+0000","last_became_peered":"2024-03-24T10:35:51.637846+0000","last_unstale":"2024-03-24T10:35:52.994549+0000","last_undegraded":"2024-03-24T10:35:52.994549+0000","last_fullsized":"2024-03-24T10:35:52.994549+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:42.375488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631281+0000","last_change":"2024-03-24T10:35:50.631866+0000","last_active":"2024-03-24T10:35:52.631281+0000","last_peered":"2024-03-24T10:35:52.631281+0000","last_clean":"2024-03-24T10:35:52.631281+0000","last_became_active":"2024-03-24T10:35:50.631578+0000","last_became_peered":"2024-03-24T10:35:50.631578+0000","last_unstale":"2024-03-24T10:35:52.631281+0000","last_undegraded":"2024-03-24T10:35:52.631281+0000","last_fullsized":"2024-03-24T10:35:52.631281+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:11.009937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994305+0000","last_change":"2024-03-24T10:35:51.629833+0000","last_active":"2024-03-24T10:35:52.994305+0000","last_peered":"2024-03-24T10:35:52.994305+0000","last_clean":"2024-03-24T10:35:52.994305+0000","last_became_active":"2024-03-24T10:35:51.629519+0000","last_became_peered":"2024-03-24T10:35:51.629519+0000","last_unstale":"2024-03-24T10:35:52.994305+0000","last_undegraded":"2024-03-24T10:35:52.994305+0000","last_fullsized":"2024-03-24T10:35:52.994305+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:02.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633555+0000","last_change":"2024-03-24T10:35:50.629954+0000","last_active":"2024-03-24T10:35:52.633555+0000","last_peered":"2024-03-24T10:35:52.633555+0000","last_clean":"2024-03-24T10:35:52.633555+0000","last_became_active":"2024-03-24T10:35:50.629658+0000","last_became_peered":"2024-03-24T10:35:50.629658+0000","last_unstale":"2024-03-24T10:35:52.633555+0000","last_undegraded":"2024-03-24T10:35:52.633555+0000","last_fullsized":"2024-03-24T10:35:52.633555+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:35:46.728350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993464+0000","last_change":"2024-03-24T10:35:51.632406+0000","last_active":"2024-03-24T10:35:52.993464+0000","last_peered":"2024-03-24T10:35:52.993464+0000","last_clean":"2024-03-24T10:35:52.993464+0000","last_became_active":"2024-03-24T10:35:51.632090+0000","last_became_peered":"2024-03-24T10:35:51.632090+0000","last_unstale":"2024-03-24T10:35:52.993464+0000","last_undegraded":"2024-03-24T10:35:52.993464+0000","last_fullsized":"2024-03-24T10:35:52.993464+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:06.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993103+0000","last_change":"2024-03-24T10:35:50.635109+0000","last_active":"2024-03-24T10:35:52.993103+0000","last_peered":"2024-03-24T10:35:52.993103+0000","last_clean":"2024-03-24T10:35:52.993103+0000","last_became_active":"2024-03-24T10:35:50.634804+0000","last_became_peered":"2024-03-24T10:35:50.634804+0000","last_unstale":"2024-03-24T10:35:52.993103+0000","last_undegraded":"2024-03-24T10:35:52.993103+0000","last_fullsized":"2024-03-24T10:35:52.993103+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:26.825292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631231+0000","last_change":"2024-03-24T10:35:51.641976+0000","last_active":"2024-03-24T10:35:52.631231+0000","last_peered":"2024-03-24T10:35:52.631231+0000","last_clean":"2024-03-24T10:35:52.631231+0000","last_became_active":"2024-03-24T10:35:51.641670+0000","last_became_peered":"2024-03-24T10:35:51.641670+0000","last_unstale":"2024-03-24T10:35:52.631231+0000","last_undegraded":"2024-03-24T10:35:52.631231+0000","last_fullsized":"2024-03-24T10:35:52.631231+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:39.341896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634392+0000","last_change":"2024-03-24T10:35:50.630807+0000","last_active":"2024-03-24T10:35:52.634392+0000","last_peered":"2024-03-24T10:35:52.634392+0000","last_clean":"2024-03-24T10:35:52.634392+0000","last_became_active":"2024-03-24T10:35:50.630512+0000","last_became_peered":"2024-03-24T10:35:50.630512+0000","last_unstale":"2024-03-24T10:35:52.634392+0000","last_undegraded":"2024-03-24T10:35:52.634392+0000","last_fullsized":"2024-03-24T10:35:52.634392+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:40:45.934135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994358+0000","last_change":"2024-03-24T10:35:51.647826+0000","last_active":"2024-03-24T10:35:52.994358+0000","last_peered":"2024-03-24T10:35:52.994358+0000","last_clean":"2024-03-24T10:35:52.994358+0000","last_became_active":"2024-03-24T10:35:51.647462+0000","last_became_peered":"2024-03-24T10:35:51.647462+0000","last_unstale":"2024-03-24T10:35:52.994358+0000","last_undegraded":"2024-03-24T10:35:52.994358+0000","last_fullsized":"2024-03-24T10:35:52.994358+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:39.341896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994250+0000","last_change":"2024-03-24T10:35:50.638566+0000","last_active":"2024-03-24T10:35:52.994250+0000","last_peered":"2024-03-24T10:35:52.994250+0000","last_clean":"2024-03-24T10:35:52.994250+0000","last_became_active":"2024-03-24T10:35:50.638280+0000","last_became_peered":"2024-03-24T10:35:50.638280+0000","last_unstale":"2024-03-24T10:35:52.994250+0000","last_undegraded":"2024-03-24T10:35:52.994250+0000","last_fullsized":"2024-03-24T10:35:52.994250+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:55.032277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.631325+0000","last_change":"2024-03-24T10:35:51.633978+0000","last_active":"2024-03-24T10:35:52.631325+0000","last_peered":"2024-03-24T10:35:52.631325+0000","last_clean":"2024-03-24T10:35:52.631325+0000","last_became_active":"2024-03-24T10:35:51.633706+0000","last_became_peered":"2024-03-24T10:35:51.633706+0000","last_unstale":"2024-03-24T10:35:52.631325+0000","last_undegraded":"2024-03-24T10:35:52.631325+0000","last_fullsized":"2024-03-24T10:35:52.631325+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:27:04.075934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995376+0000","last_change":"2024-03-24T10:35:50.631008+0000","last_active":"2024-03-24T10:35:52.995376+0000","last_peered":"2024-03-24T10:35:52.995376+0000","last_clean":"2024-03-24T10:35:52.995376+0000","last_became_active":"2024-03-24T10:35:50.630722+0000","last_became_peered":"2024-03-24T10:35:50.630722+0000","last_unstale":"2024-03-24T10:35:52.995376+0000","last_undegraded":"2024-03-24T10:35:52.995376+0000","last_fullsized":"2024-03-24T10:35:52.995376+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:19.890853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633733+0000","last_change":"2024-03-24T10:35:51.638152+0000","last_active":"2024-03-24T10:35:52.633733+0000","last_peered":"2024-03-24T10:35:52.633733+0000","last_clean":"2024-03-24T10:35:52.633733+0000","last_became_active":"2024-03-24T10:35:51.637901+0000","last_became_peered":"2024-03-24T10:35:51.637901+0000","last_unstale":"2024-03-24T10:35:52.633733+0000","last_undegraded":"2024-03-24T10:35:52.633733+0000","last_fullsized":"2024-03-24T10:35:52.633733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:11:35.608754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993247+0000","last_change":"2024-03-24T10:35:50.636330+0000","last_active":"2024-03-24T10:35:52.993247+0000","last_peered":"2024-03-24T10:35:52.993247+0000","last_clean":"2024-03-24T10:35:52.993247+0000","last_became_active":"2024-03-24T10:35:50.636035+0000","last_became_peered":"2024-03-24T10:35:50.636035+0000","last_unstale":"2024-03-24T10:35:52.993247+0000","last_undegraded":"2024-03-24T10:35:52.993247+0000","last_fullsized":"2024-03-24T10:35:52.993247+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:41:12.007000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993284+0000","last_change":"2024-03-24T10:35:51.648077+0000","last_active":"2024-03-24T10:35:52.993284+0000","last_peered":"2024-03-24T10:35:52.993284+0000","last_clean":"2024-03-24T10:35:52.993284+0000","last_became_active":"2024-03-24T10:35:51.647776+0000","last_became_peered":"2024-03-24T10:35:51.647776+0000","last_unstale":"2024-03-24T10:35:52.993284+0000","last_undegraded":"2024-03-24T10:35:52.993284+0000","last_fullsized":"2024-03-24T10:35:52.993284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:26.341866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633753+0000","last_change":"2024-03-24T10:35:50.628958+0000","last_active":"2024-03-24T10:35:52.633753+0000","last_peered":"2024-03-24T10:35:52.633753+0000","last_clean":"2024-03-24T10:35:52.633753+0000","last_became_active":"2024-03-24T10:35:50.628661+0000","last_became_peered":"2024-03-24T10:35:50.628661+0000","last_unstale":"2024-03-24T10:35:52.633753+0000","last_undegraded":"2024-03-24T10:35:52.633753+0000","last_fullsized":"2024-03-24T10:35:52.633753+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:27:03.071254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994924+0000","last_change":"2024-03-24T10:35:51.642843+0000","last_active":"2024-03-24T10:35:52.994924+0000","last_peered":"2024-03-24T10:35:52.994924+0000","last_clean":"2024-03-24T10:35:52.994924+0000","last_became_active":"2024-03-24T10:35:51.642497+0000","last_became_peered":"2024-03-24T10:35:51.642497+0000","last_unstale":"2024-03-24T10:35:52.994924+0000","last_undegraded":"2024-03-24T10:35:52.994924+0000","last_fullsized":"2024-03-24T10:35:52.994924+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:35:47.733029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.633651+0000","last_change":"2024-03-24T10:35:50.627406+0000","last_active":"2024-03-24T10:35:52.633651+0000","last_peered":"2024-03-24T10:35:52.633651+0000","last_clean":"2024-03-24T10:35:52.633651+0000","last_became_active":"2024-03-24T10:35:50.627116+0000","last_became_peered":"2024-03-24T10:35:50.627116+0000","last_unstale":"2024-03-24T10:35:52.633651+0000","last_undegraded":"2024-03-24T10:35:52.633651+0000","last_fullsized":"2024-03-24T10:35:52.633651+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:38.337216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993425+0000","last_change":"2024-03-24T10:35:51.640902+0000","last_active":"2024-03-24T10:35:52.993425+0000","last_peered":"2024-03-24T10:35:52.993425+0000","last_clean":"2024-03-24T10:35:52.993425+0000","last_became_active":"2024-03-24T10:35:51.640598+0000","last_became_peered":"2024-03-24T10:35:51.640598+0000","last_unstale":"2024-03-24T10:35:52.993425+0000","last_undegraded":"2024-03-24T10:35:52.993425+0000","last_fullsized":"2024-03-24T10:35:52.993425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:36.246303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993690+0000","last_change":"2024-03-24T10:35:50.635584+0000","last_active":"2024-03-24T10:35:52.993690+0000","last_peered":"2024-03-24T10:35:52.993690+0000","last_clean":"2024-03-24T10:35:52.993690+0000","last_became_active":"2024-03-24T10:35:50.635280+0000","last_became_peered":"2024-03-24T10:35:50.635280+0000","last_unstale":"2024-03-24T10:35:52.993690+0000","last_undegraded":"2024-03-24T10:35:52.993690+0000","last_fullsized":"2024-03-24T10:35:52.993690+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:54.961171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.634177+0000","last_change":"2024-03-24T10:35:51.643650+0000","last_active":"2024-03-24T10:35:52.634177+0000","last_peered":"2024-03-24T10:35:52.634177+0000","last_clean":"2024-03-24T10:35:52.634177+0000","last_became_active":"2024-03-24T10:35:51.639802+0000","last_became_peered":"2024-03-24T10:35:51.639802+0000","last_unstale":"2024-03-24T10:35:52.634177+0000","last_undegraded":"2024-03-24T10:35:52.634177+0000","last_fullsized":"2024-03-24T10:35:52.634177+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:57:19.073753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.994734+0000","last_change":"2024-03-24T10:35:50.631769+0000","last_active":"2024-03-24T10:35:52.994734+0000","last_peered":"2024-03-24T10:35:52.994734+0000","last_clean":"2024-03-24T10:35:52.994734+0000","last_became_active":"2024-03-24T10:35:50.631453+0000","last_became_peered":"2024-03-24T10:35:50.631453+0000","last_unstale":"2024-03-24T10:35:52.994734+0000","last_undegraded":"2024-03-24T10:35:52.994734+0000","last_fullsized":"2024-03-24T10:35:52.994734+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:23.367988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993609+0000","last_change":"2024-03-24T10:35:51.629419+0000","last_active":"2024-03-24T10:35:52.993609+0000","last_peered":"2024-03-24T10:35:52.993609+0000","last_clean":"2024-03-24T10:35:52.993609+0000","last_became_active":"2024-03-24T10:35:51.629052+0000","last_became_peered":"2024-03-24T10:35:51.629052+0000","last_unstale":"2024-03-24T10:35:52.993609+0000","last_undegraded":"2024-03-24T10:35:52.993609+0000","last_fullsized":"2024-03-24T10:35:52.993609+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:50.608329+0000","last_clean_scrub_stamp":"2024-03-24T10:35:50.608329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:46:14.050698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"50'1","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.993551+0000","last_change":"2024-03-24T10:35:50.636812+0000","last_active":"2024-03-24T10:35:52.993551+0000","last_peered":"2024-03-24T10:35:52.993551+0000","last_clean":"2024-03-24T10:35:52.993551+0000","last_became_active":"2024-03-24T10:35:50.636520+0000","last_became_peered":"2024-03-24T10:35:50.636520+0000","last_unstale":"2024-03-24T10:35:52.993551+0000","last_undegraded":"2024-03-24T10:35:52.993551+0000","last_fullsized":"2024-03-24T10:35:52.993551+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:35:49.603649+0000","last_clean_scrub_stamp":"2024-03-24T10:35:49.603649+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:20:59.508106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995039+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:52.995039+0000","last_peered":"2024-03-24T10:35:52.995039+0000","last_clean":"2024-03-24T10:35:52.995039+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:52.995039+0000","last_undegraded":"2024-03-24T10:35:52.995039+0000","last_fullsized":"2024-03-24T10:35:52.995039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343652,"statfs":{"total":96636764160,"available":96607899648,"internally_reserved":0,"allocated":1196032,"data_stored":837458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":246966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":247548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":248600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":606208,"data_stored":248578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":602112,"data_stored":247126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1196032,"data_stored":837390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:35:59.662 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T10:35:59.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T10:35:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.806+0000 7fd3f16c0640 1 Processor -- start 2024-03-24T10:35:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.806+0000 7fd3f16c0640 1 -- start start 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3f16c0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 0x7fd3ec0d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3f16c0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd3ec0cfae0 0x7fd3ec0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3f16c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3ec0b0b70 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3f16c0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ec0d1d60 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3f16c0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ec0b0de0 con 0x7fd3ec15e390 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd3ec0cfae0 0x7fd3ec0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3ea7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd3ec0b2ed0 0x7fd3ec0b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49958/0 (socket says 172.21.15.1:49958) 2024-03-24T10:35:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3ea7fc640 1 -- 172.21.15.1:0/3048074161 learned_addr learned my addr 172.21.15.1:0/3048074161 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eaffd640 1 --2- 172.21.15.1:0/3048074161 >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 0x7fd3ec0d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eb7fe640 1 -- 172.21.15.1:0/3048074161 >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 msgr2=0x7fd3ec0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eb7fe640 1 --2- 172.21.15.1:0/3048074161 >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 0x7fd3ec0d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eb7fe640 1 -- 172.21.15.1:0/3048074161 >> v1:172.21.15.1:6789/0 conn(0x7fd3ec0b2ed0 legacy=0x7fd3ec0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eb7fe640 1 -- 172.21.15.1:0/3048074161 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec15e770 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3e9ffb640 1 -- 172.21.15.1:0/3048074161 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133209186 0 0) 0x7fd3ec0b0b70 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.807+0000 7fd3eaffd640 1 --2- 172.21.15.1:0/3048074161 >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 0x7fd3ec0d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:59.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.808+0000 7fd3eb7fe640 1 --2- 172.21.15.1:0/3048074161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cfae0 0x7fd3ec0d2850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec08c020 tx=0x7fd3dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161d2b7c51af9fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.808+0000 7fd3e9ffb640 1 -- 172.21.15.1:0/3048074161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc02cc80 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.808+0000 7fd3e9ffb640 1 -- 172.21.15.1:0/3048074161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3dc02cde0 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.808+0000 7fd3e9ffb640 1 -- 172.21.15.1:0/3048074161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc034660 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3048074161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cfae0 msgr2=0x7fd3ec0d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3048074161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cfae0 0x7fd3ec0d2850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec08c020 tx=0x7fd3dc02bed0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3048074161 shutdown_connections 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3048074161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cfae0 0x7fd3ec0d2850 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3048074161 >> v2:172.21.15.3:3300/0 conn(0x7fd3ec15e390 0x7fd3ec0d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3048074161 >> 172.21.15.1:0/3048074161 conn(0x7fd3ec0d3ee0 msgr2=0x7fd3ec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3048074161 shutdown_connections 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.809+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3048074161 wait complete. 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.811+0000 7fd3f16c0640 1 Processor -- start 2024-03-24T10:35:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.811+0000 7fd3f16c0640 1 -- start start 2024-03-24T10:35:59.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3ec0cfae0 0x7fd3ec1f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ec15e390 0x7fd3ec0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec0cc410 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec0cbf00 con 0x7fd3ec15e390 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3f16c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ec0b4b70 con 0x7fd3ec0cfae0 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.812+0000 7fd3eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38694/0 (socket says 172.21.15.1:38694) 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 -- 172.21.15.1:0/3703645185 learned_addr learned my addr 172.21.15.1:0/3703645185 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eb7fe640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ec15e390 0x7fd3ec0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3ea7fc640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3ec0cfae0 0x7fd3ec1f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 -- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3ec0cfae0 msgr2=0x7fd3ec1f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3ec0cfae0 0x7fd3ec1f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 -- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ec15e390 msgr2=0x7fd3ec0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ec15e390 0x7fd3ec0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 -- 172.21.15.1:0/3703645185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc009a70 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3eaffd640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c7f0 tx=0x7fd3e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4007820 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e4007980 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec1f3590 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4007cf0 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.813+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec0b4520 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.814+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3e40113f0 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.814+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ec0b4380 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.815+0000 7fd3d37fe640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 0x7fd3d806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.815+0000 7fd3ea7fc640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 0x7fd3d806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.815+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fd3e408a490 con 0x7fd3ec0b2ed0 2024-03-24T10:35:59.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.816+0000 7fd3ea7fc640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 0x7fd3d806dd00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0cc7c0 tx=0x7fd3e0007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:59.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:35:59.819+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd3ec0b4380 con 0x7fd3ec0b2ed0 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.084+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd3ec0b4380 con 0x7fd3ec0b2ed0 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.086+0000 7fd3d37fe640 1 -- 172.21.15.1:0/3703645185 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7fd3ec0b4380 con 0x7fd3ec0b2ed0 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:36:00.087 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T10:36:00.088 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:36:00.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 msgr2=0x7fd3d806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 0x7fd3d806dd00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0cc7c0 tx=0x7fd3e0007b50 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 msgr2=0x7fd3ec0cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c7f0 tx=0x7fd3e400ccb0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 shutdown_connections 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd3d806b840 0x7fd3d806dd00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ec15e390 0x7fd3ec0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd3ec0cfae0 0x7fd3ec1f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 --2- 172.21.15.1:0/3703645185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0b2ed0 0x7fd3ec0cb9c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 >> 172.21.15.1:0/3703645185 conn(0x7fd3ec0d3ee0 msgr2=0x7fd3ec0b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 shutdown_connections 2024-03-24T10:36:00.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.088+0000 7fd3f16c0640 1 -- 172.21.15.1:0/3703645185 wait complete. 2024-03-24T10:36:00.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:00.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 Processor -- start 2024-03-24T10:36:00.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 -- start start 2024-03-24T10:36:00.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f897415e390 0x7f89740d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 0x7f89740d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897415e770 con 0x7f89740d1000 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f89740b3190 con 0x7f897415e390 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f8979b3e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f89740d4e40 con 0x7f89740b4ba0 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.201+0000 7f89737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f897415e390 0x7f89740d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f897415e390 0x7f89740d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38706/0 (socket says 172.21.15.1:38706) 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 -- 172.21.15.1:0/846688959 learned_addr learned my addr 172.21.15.1:0/846688959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 -- 172.21.15.1:0/846688959 >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 msgr2=0x7f89740d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f8973fff640 1 --2- 172.21.15.1:0/846688959 >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 0x7f89740d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 --2- 172.21.15.1:0/846688959 >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 0x7f89740d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 -- 172.21.15.1:0/846688959 >> v1:172.21.15.3:6789/0 conn(0x7f89740d1000 legacy=0x7f89740cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f89737fe640 1 -- 172.21.15.1:0/846688959 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89740d1e40 con 0x7f897415e390 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.202+0000 7f8973fff640 1 --2- 172.21.15.1:0/846688959 >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 0x7f89740d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:00.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f89737fe640 1 --2- 172.21.15.1:0/846688959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f897415e390 0x7f89740d24a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f897408e5b0 tx=0x7f896402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5bb9ce089860897 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f89727fc640 1 -- 172.21.15.1:0/846688959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896402c960 con 0x7f897415e390 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f89727fc640 1 -- 172.21.15.1:0/846688959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896402cac0 con 0x7f897415e390 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f89727fc640 1 -- 172.21.15.1:0/846688959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896402ce50 con 0x7f897415e390 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f8979b3e640 1 -- 172.21.15.1:0/846688959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f897415e390 msgr2=0x7f89740d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.203+0000 7f8979b3e640 1 --2- 172.21.15.1:0/846688959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f897415e390 0x7f89740d24a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f897408e5b0 tx=0x7f896402bdc0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 -- 172.21.15.1:0/846688959 shutdown_connections 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 --2- 172.21.15.1:0/846688959 >> v2:172.21.15.3:3300/0 conn(0x7f89740b4ba0 0x7f89740d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 --2- 172.21.15.1:0/846688959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f897415e390 0x7f89740d24a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 -- 172.21.15.1:0/846688959 >> 172.21.15.1:0/846688959 conn(0x7f89740d3ec0 msgr2=0x7f89740d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 -- 172.21.15.1:0/846688959 shutdown_connections 2024-03-24T10:36:00.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.204+0000 7f8979b3e640 1 -- 172.21.15.1:0/846688959 wait complete. 2024-03-24T10:36:00.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.205+0000 7f8979b3e640 1 Processor -- start 2024-03-24T10:36:00.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.205+0000 7f8979b3e640 1 -- start start 2024-03-24T10:36:00.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f89740d1000 0x7f89740cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f897415e390 0x7f89740cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f89740b6ae0 con 0x7f89740b4ba0 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8974152860 con 0x7f897415e390 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f89740cc320 con 0x7f89740d1000 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38716/0 (socket says 172.21.15.1:38716) 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 -- 172.21.15.1:0/184034567 learned_addr learned my addr 172.21.15.1:0/184034567 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8972ffd640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f89740d1000 0x7f89740cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8973fff640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f897415e390 0x7f89740cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 -- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f89740d1000 msgr2=0x7f89740cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f89740d1000 0x7f89740cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 -- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f897415e390 msgr2=0x7f89740cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f897415e390 0x7f89740cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 -- 172.21.15.1:0/184034567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8964009b30 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f89737fe640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f896402c3a0 tx=0x7f8964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896402ce30 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8964040470 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89740b6d30 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.206+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964040800 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.207+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89740b72f0 con 0x7f89740b4ba0 2024-03-24T10:36:00.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.208+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897404f4e0 con 0x7f89740b4ba0 2024-03-24T10:36:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.208+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8964040960 con 0x7f89740b4ba0 2024-03-24T10:36:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.209+0000 7f896bfff640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.209+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f89640acd80 con 0x7f89740b4ba0 2024-03-24T10:36:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.209+0000 7f8972ffd640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.209+0000 7f8972ffd640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 0x7f895406dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8974152b50 tx=0x7f895c007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.217+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89640778e0 con 0x7f89740b4ba0 2024-03-24T10:36:00.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.434+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f897415e770 con 0x7f89740b4ba0 2024-03-24T10:36:00.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.435+0000 7f896bfff640 1 -- 172.21.15.1:0/184034567 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f897415e770 con 0x7f89740b4ba0 2024-03-24T10:36:00.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:00.435 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:36:00.436 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:36:00.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 msgr2=0x7f895406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 0x7f895406dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8974152b50 tx=0x7f895c007e30 comp rx=0 tx=0).stop 2024-03-24T10:36:00.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 msgr2=0x7f8974152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f896402c3a0 tx=0x7f8964004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:00.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 shutdown_connections 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f897415e390 0x7f89740cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f89740d1000 0x7f89740cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 --2- 172.21.15.1:0/184034567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89740b4ba0 0x7f8974152320 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 >> 172.21.15.1:0/184034567 conn(0x7f89740d3ec0 msgr2=0x7f89740cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 shutdown_connections 2024-03-24T10:36:00.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.437+0000 7f8979b3e640 1 -- 172.21.15.1:0/184034567 wait complete. 2024-03-24T10:36:00.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:35:50.983041+0000', 'modified': '2024-03-24T10:35:52.981728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5982}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5982': {'gid': 5982, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/1119214087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 1119214087}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 1119214087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5982, 'qdb_cluster': [5982]} 2024-03-24T10:36:00.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T10:36:00.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T10:36:00.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.954+0000 7f9da2543640 1 Processor -- start 2024-03-24T10:36:00.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.954+0000 7f9da2543640 1 -- start start 2024-03-24T10:36:00.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da2543640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da2543640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d9c15e770 con 0x7f9d9c0d1000 2024-03-24T10:36:00.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da2543640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d9c0d4e40 con 0x7f9d9c15e390 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da2543640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9d9c0b3190 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da0ae0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9d9c0d1000 0x7f9d9c0b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40834/0 (socket says 172.21.15.1:40834) 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9da0ae0640 1 -- 172.21.15.1:0/4162093521 learned_addr learned my addr 172.21.15.1:0/4162093521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/4162093521 >> v2:172.21.15.1:3300/0 conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54390/0 (socket says 172.21.15.1:54390) 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/4162093521 >> v1:172.21.15.3:6789/0 conn(0x7f9d9c15e390 legacy=0x7f9d9c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/4162093521 >> v1:172.21.15.1:6789/0 conn(0x7f9d9c0d1000 legacy=0x7f9d9c0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/4162093521 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9c0d1e40 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.956+0000 7f9d9affd640 1 -- 172.21.15.1:0/4162093521 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565150495 0 0) 0x7f9d94001490 con 0x7f9d9c0d1000 2024-03-24T10:36:00.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.957+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/4162093521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9d900097a0 tx=0x7f9d9002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91e05fe5fc1c224 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.957+0000 7f9d9affd640 1 -- 172.21.15.1:0/4162093521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9002e070 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.957+0000 7f9d9affd640 1 -- 172.21.15.1:0/4162093521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d900315a0 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.957+0000 7f9d9affd640 1 -- 172.21.15.1:0/4162093521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90031930 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 -- 172.21.15.1:0/4162093521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 msgr2=0x7f9d9c0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 --2- 172.21.15.1:0/4162093521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9d900097a0 tx=0x7f9d9002b840 comp rx=0 tx=0).stop 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 -- 172.21.15.1:0/4162093521 shutdown_connections 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 --2- 172.21.15.1:0/4162093521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c0b3b90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 -- 172.21.15.1:0/4162093521 >> 172.21.15.1:0/4162093521 conn(0x7f9d9c0d3ec0 msgr2=0x7f9d9c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 -- 172.21.15.1:0/4162093521 shutdown_connections 2024-03-24T10:36:00.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.958+0000 7f9da2543640 1 -- 172.21.15.1:0/4162093521 wait complete. 2024-03-24T10:36:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.959+0000 7f9da2543640 1 Processor -- start 2024-03-24T10:36:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.959+0000 7f9da2543640 1 -- start start 2024-03-24T10:36:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 0x7f9d9c155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d9c15e390 0x7f9d9c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c09f5c0 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c1e3730 con 0x7f9d9c15e390 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da2543640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9c1f4160 con 0x7f9d9c0d1000 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54406/0 (socket says 172.21.15.1:54406) 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/3595423053 learned_addr learned my addr 172.21.15.1:0/3595423053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d9c15e390 0x7f9d9c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 0x7f9d9c155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 msgr2=0x7f9d9c155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 0x7f9d9c155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d9c15e390 msgr2=0x7f9d9c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d9c15e390 0x7f9d9c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/3595423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d90009400 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 0x7f9d9c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.960+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9d900098c0 tx=0x7f9d900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.961+0000 7f9d98ff9640 1 -- 172.21.15.1:0/3595423053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90032980 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.961+0000 7f9d98ff9640 1 -- 172.21.15.1:0/3595423053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d90032ae0 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.961+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9c09ea40 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.961+0000 7f9d98ff9640 1 -- 172.21.15.1:0/3595423053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90032e90 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.961+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d9c09edd0 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.962+0000 7f9d98ff9640 1 -- 172.21.15.1:0/3595423053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9d9004d3f0 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.963+0000 7f9d98ff9640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 0x7f9d8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.963+0000 7f9d98ff9640 1 -- 172.21.15.1:0/3595423053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f9d900abf00 con 0x7f9d9c0d2bb0 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.963+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 0x7f9d8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.963+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 0x7f9d8806dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007fd0 tx=0x7f9d8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.963+0000 7f9da51f0740 1 Processor -- start 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 -- start start 2024-03-24T10:36:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3979f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3976faf0 0x560e3979f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e396e3180 con 0x560e3979c610 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x560e396e06b0 con 0x560e3976faf0 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x560e396dbe90 con 0x560e3979c9f0 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9d9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9d9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54412/0 (socket says 172.21.15.1:54412) 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.964+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1955235060 learned_addr learned my addr 172.21.15.1:0/1955235060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3979f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 msgr2=0x560e3979f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3976faf0 0x560e3979f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3979f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3976faf0 msgr2=0x560e3979f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3976faf0 0x560e3979f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1955235060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e39698d80 con 0x560e3979c610 2024-03-24T10:36:00.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c005100 tx=0x7f9d8c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85cf17dda751951 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d81ffb640 1 -- 172.21.15.1:0/1955235060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0035f0 con 0x560e3979c610 2024-03-24T10:36:00.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d81ffb640 1 -- 172.21.15.1:0/1955235060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c003750 con 0x560e3979c610 2024-03-24T10:36:00.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.965+0000 7f9d81ffb640 1 -- 172.21.15.1:0/1955235060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c003ac0 con 0x560e3979c610 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 -- 172.21.15.1:0/1955235060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 msgr2=0x560e3976f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c005100 tx=0x7f9d8c011000 comp rx=0 tx=0).stop 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 -- 172.21.15.1:0/1955235060 shutdown_connections 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3976faf0 0x560e3979f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3979f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1955235060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e3976f5b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 -- 172.21.15.1:0/1955235060 >> 172.21.15.1:0/1955235060 conn(0x560e39771820 msgr2=0x560e397ba3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 -- 172.21.15.1:0/1955235060 shutdown_connections 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.966+0000 7f9da51f0740 1 -- 172.21.15.1:0/1955235060 wait complete. 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.967+0000 7f9da51f0740 1 -- start start 2024-03-24T10:36:00.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.967+0000 7f9da51f0740 1 Processor -- start 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e397a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39799520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e39680d50 con 0x560e3979a5c0 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x560e393af310 con 0x560e3979c610 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x560e396618f0 con 0x560e3979c9f0 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da51f0740 10 client.0 _ll_register_callbacks cb 0x560e393ecac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da0ae0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da0ae0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54422/0 (socket says 172.21.15.1:54422) 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9da0ae0640 1 -- 172.21.15.1:0/4176196833 learned_addr learned my addr 172.21.15.1:0/4176196833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.968+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39799520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e397a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 -- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 msgr2=0x560e39799520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39799520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 -- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 msgr2=0x560e397a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e397a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 -- 172.21.15.1:0/4176196833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c00f040 con 0x560e3979a5c0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9d94004df0 tx=0x7f9d94007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2e659a0167f09f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94013570 con 0x560e3979a5c0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f9d9400c780 con 0x560e3979a5c0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e393db6c0 con 0x560e3979a5c0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d9400c940 con 0x560e3979a5c0 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da51f0740 10 client.6141 fetch_fsmap learned FSMap version 25 2024-03-24T10:36:00.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x560e396da510 con 0x560e3979a5c0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d94023020 con 0x560e3979a5c0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.969+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9d9400caa0 con 0x560e3979a5c0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 10 client.6141 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 10 client.6141 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 20 client.6141 populate_metadata read hostname 'smithi001' 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e397d55d0 0x560e397d79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 4 client.6141 mds_command: new command op to 5982 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 --> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e3942c270 con 0x560e397d55d0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e397d55d0 0x560e397d79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e397d55d0 0x560e397d79c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a89d5a8042ce6325 server_cookie=c3099af8cb174de1 in_seq=0 out_seq=0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.970+0000 7f9d7affd640 10 client.6141 ms_handle_connect on v2:172.21.15.1:6836/1119214087 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9d7affd640 1 -- 172.21.15.1:0/4176196833 <== mds.0 v2:172.21.15.1:6836/1119214087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e3942c270 con 0x560e397d55d0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9d7affd640 10 client.6141 handle_command_reply: tid=0 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 client.6141 shutdown 2024-03-24T10:36:00.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 msgr2=0x560e39799a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9d94004df0 tx=0x7f9d94007430 comp rx=0 tx=0).stop 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 shutdown_connections 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e397d55d0 0x560e397d79c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e39799a60 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39799520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 --2- 172.21.15.1:0/4176196833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e397a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 >> 172.21.15.1:0/4176196833 conn(0x560e3976faf0 msgr2=0x560e397763e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9d7affd640 0 client.6141 ms_handle_reset on v2:172.21.15.1:6836/1119214087 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 shutdown_connections 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 1 -- 172.21.15.1:0/4176196833 wait complete. 2024-03-24T10:36:00.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.971+0000 7f9da51f0740 20 client.6141 trim_cache size 0 max 16384 2024-03-24T10:36:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.989+0000 7f9da51f0740 1 Processor -- start 2024-03-24T10:36:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.989+0000 7f9da51f0740 1 -- start start 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e3976e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3976e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e397a2660 con 0x560e3979a5c0 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x560e39775590 con 0x560e3979c610 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x560e397a1230 con 0x560e3979c9f0 2024-03-24T10:36:00.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54436/0 (socket says 172.21.15.1:54436) 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1082331325 learned_addr learned my addr 172.21.15.1:0/1082331325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e3976e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3976e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 msgr2=0x560e3976e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3976e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 msgr2=0x560e3976e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e3976e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 -- 172.21.15.1:0/1082331325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e39676a80 con 0x560e3979a5c0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3976e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.990+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0050d0 tx=0x7f9d8c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e06927daade49e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9d817fa640 1 -- 172.21.15.1:0/1082331325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c028030 con 0x560e3979a5c0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9d817fa640 1 -- 172.21.15.1:0/1082331325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c029040 con 0x560e3979a5c0 2024-03-24T10:36:00.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9d817fa640 1 -- 172.21.15.1:0/1082331325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c030680 con 0x560e3979a5c0 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 -- 172.21.15.1:0/1082331325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 msgr2=0x560e397a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0050d0 tx=0x7f9d8c005000 comp rx=0 tx=0).stop 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 -- 172.21.15.1:0/1082331325 shutdown_connections 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e3976e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979c610 0x560e3976e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 --2- 172.21.15.1:0/1082331325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979a5c0 0x560e397a29e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:00.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 -- 172.21.15.1:0/1082331325 >> 172.21.15.1:0/1082331325 conn(0x560e3976faf0 msgr2=0x560e397a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:00.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 -- 172.21.15.1:0/1082331325 shutdown_connections 2024-03-24T10:36:00.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.991+0000 7f9da51f0740 1 -- 172.21.15.1:0/1082331325 wait complete. 2024-03-24T10:36:00.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.992+0000 7f9da51f0740 1 -- start start 2024-03-24T10:36:00.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.992+0000 7f9da51f0740 1 Processor -- start 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979a5c0 0x560e398228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39832760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e3979b360 con 0x560e3979c610 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x560e39823bb0 con 0x560e3979a5c0 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x560e3979c030 con 0x560e3979c9f0 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da51f0740 10 client.0 _ll_register_callbacks cb 0x7f9d94001e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:36:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54442/0 (socket says 172.21.15.1:54442) 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/2555500279 learned_addr learned my addr 172.21.15.1:0/2555500279 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979a5c0 0x560e398228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9da0ae0640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39832760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 msgr2=0x560e39832760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39832760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979a5c0 msgr2=0x560e398228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979a5c0 0x560e398228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 -- 172.21.15.1:0/2555500279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e39698d80 con 0x560e3979c610 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d9b7fe640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9d90076a50 tx=0x7f9d9007adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebdbc441a2feae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d7affd640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.993+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90082d90 con 0x560e3979c610 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f9d90082900 con 0x560e3979c610 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d8c003750 con 0x560e3979c610 2024-03-24T10:36:01.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d90082ae0 con 0x560e3979c610 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 10 client.6147 fetch_fsmap learned FSMap version 25 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x560e3969c7e0 con 0x560e3979c610 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c003750 con 0x560e3979c610 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9d90079a40 con 0x560e3979c610 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 10 client.6147 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 10 client.6147 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 20 client.6147 populate_metadata read hostname 'smithi001' 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e3976e450 0x560e3976e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 4 client.6147 mds_command: new command op to 5982 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 --> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] -- command(tid 0: {"prefix": "status"}) -- 0x560e39418610 con 0x560e3976e450 2024-03-24T10:36:01.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e3976e450 0x560e3976e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d7affd640 10 client.6147 ms_handle_connect on v2:172.21.15.1:6836/1119214087 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.994+0000 7f9d9bfff640 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e3976e450 0x560e3976e830 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54fac867650c422f server_cookie=35dc463a499cf6f5 in_seq=0 out_seq=0 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9d7affd640 1 -- 172.21.15.1:0/2555500279 <== mds.0 v2:172.21.15.1:6836/1119214087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x560e39418610 con 0x560e3976e450 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9d7affd640 10 client.6147 handle_command_reply: tid=0 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 client.6147 shutdown 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 msgr2=0x560e39822e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9d90076a50 tx=0x7f9d9007adb0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 shutdown_connections 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:6836/1119214087,v1:172.21.15.1:6837/1119214087] conn(0x560e3976e450 0x560e3976e830 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x560e3979c9f0 0x560e39832760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e3979c610 0x560e39822e00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 --2- 172.21.15.1:0/2555500279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560e3979a5c0 0x560e398228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9d7affd640 0 client.6147 ms_handle_reset on v2:172.21.15.1:6836/1119214087 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 >> 172.21.15.1:0/2555500279 conn(0x560e3976faf0 msgr2=0x560e3976a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 shutdown_connections 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 1 -- 172.21.15.1:0/2555500279 wait complete. 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.995+0000 7f9da51f0740 20 client.6147 trim_cache size 0 max 16384 2024-03-24T10:36:01.003 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "a0192ab9-51d5-4157-8f81-4a83111a7dfa", 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5982, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 9.3794972649999995, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 50, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 50, 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.657928957999999 2024-03-24T10:36:01.004 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 msgr2=0x7f9d8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 0x7f9d8806dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007fd0 tx=0x7f9d8c009040 comp rx=0 tx=0).stop 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 msgr2=0x7f9d9c09e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9d900098c0 tx=0x7f9d900046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 shutdown_connections 2024-03-24T10:36:01.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f9d8806b810 0x7f9d8806dcd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d9c15e390 0x7f9d9c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d9c0d2bb0 0x7f9d9c09e060 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 --2- 172.21.15.1:0/3595423053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f9d9c0d1000 0x7f9d9c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 >> 172.21.15.1:0/3595423053 conn(0x7f9d9c0d3ec0 msgr2=0x7f9d9c0d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 shutdown_connections 2024-03-24T10:36:01.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:00.996+0000 7f9da2543640 1 -- 172.21.15.1:0/3595423053 wait complete. 2024-03-24T10:36:01.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:36:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.109+0000 7f5af89a0640 1 Processor -- start 2024-03-24T10:36:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.109+0000 7f5af89a0640 1 -- start start 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af89a0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af89a0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 0x7f5af40b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af89a0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af415e770 con 0x7f5af415e390 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af89a0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f5af40b3190 con 0x7f5af40d1000 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af89a0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5af40d4e40 con 0x7f5af40d2bb0 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af2d9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 0x7f5af40b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:54488/0 (socket says 172.21.15.1:54488) 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 -- 172.21.15.1:0/3322036632 learned_addr learned my addr 172.21.15.1:0/3322036632 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af259c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5af415e390 0x7f5af40d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:59954/0 (socket says 172.21.15.1:59954) 2024-03-24T10:36:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af159a640 1 -- 172.21.15.1:0/3322036632 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716870923 0 0) 0x7f5ae4001110 con 0x7f5af415e390 2024-03-24T10:36:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 -- 172.21.15.1:0/3322036632 >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 msgr2=0x7f5af40b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 --2- 172.21.15.1:0/3322036632 >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 0x7f5af40b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 -- 172.21.15.1:0/3322036632 >> v1:172.21.15.3:6789/0 conn(0x7f5af415e390 legacy=0x7f5af40d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af1d9b640 1 -- 172.21.15.1:0/3322036632 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af40d1e40 con 0x7f5af40d2bb0 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.110+0000 7f5af2d9d640 1 --2- 172.21.15.1:0/3322036632 >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 0x7f5af40b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.111+0000 7f5af1d9b640 1 --2- 172.21.15.1:0/3322036632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35fb06bd0f40eff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.111+0000 7f5af159a640 1 -- 172.21.15.1:0/3322036632 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80317c0 con 0x7f5af40d2bb0 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.111+0000 7f5af159a640 1 -- 172.21.15.1:0/3322036632 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae8031920 con 0x7f5af40d2bb0 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.111+0000 7f5af159a640 1 -- 172.21.15.1:0/3322036632 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8031cb0 con 0x7f5af40d2bb0 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 -- 172.21.15.1:0/3322036632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 msgr2=0x7f5af40b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 --2- 172.21.15.1:0/3322036632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009870 tx=0x7f5ae802b860 comp rx=0 tx=0).stop 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 -- 172.21.15.1:0/3322036632 shutdown_connections 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 --2- 172.21.15.1:0/3322036632 >> v2:172.21.15.3:3300/0 conn(0x7f5af40d1000 0x7f5af40b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 --2- 172.21.15.1:0/3322036632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af40d2bb0 0x7f5af40b3b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 -- 172.21.15.1:0/3322036632 >> 172.21.15.1:0/3322036632 conn(0x7f5af40d3ec0 msgr2=0x7f5af40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 -- 172.21.15.1:0/3322036632 shutdown_connections 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.112+0000 7f5af89a0640 1 -- 172.21.15.1:0/3322036632 wait complete. 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.113+0000 7f5af89a0640 1 Processor -- start 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.113+0000 7f5af89a0640 1 -- start start 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40d2bb0 0x7f5af415a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af415e390 0x7f5af41f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5af40b7090 con 0x7f5af40d1000 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f5af415add0 con 0x7f5af40d2bb0 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af89a0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f5af415b0b0 con 0x7f5af415e390 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54458/0 (socket says 172.21.15.1:54458) 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af259c640 1 -- 172.21.15.1:0/1334060774 learned_addr learned my addr 172.21.15.1:0/1334060774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af1d9b640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40d2bb0 0x7f5af415a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af2d9d640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af415e390 0x7f5af41f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.114+0000 7f5af259c640 1 -- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af415e390 msgr2=0x7f5af41f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af259c640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af415e390 0x7f5af41f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af259c640 1 -- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40d2bb0 msgr2=0x7f5af415a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af259c640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40d2bb0 0x7f5af415a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af259c640 1 -- 172.21.15.1:0/1334060774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec009a70 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af259c640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c430 tx=0x7f5ae400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40074e0 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae4007660 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae80094d0 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4004a20 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.115+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af40b7630 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.116+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af404f4e0 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.116+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ae4004b80 con 0x7f5af40d1000 2024-03-24T10:36:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.117+0000 7f5ae2ffd640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 0x7f5ad406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.117+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f5ae4089d30 con 0x7f5af40d1000 2024-03-24T10:36:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.117+0000 7f5af1d9b640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 0x7f5ad406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:01.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.117+0000 7f5af1d9b640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 0x7f5ad406dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8036000 tx=0x7f5ae803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:01.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.124+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ae4054760 con 0x7f5af40d1000 2024-03-24T10:36:01.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.343+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5af415e770 con 0x7f5af40d1000 2024-03-24T10:36:01.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.344+0000 7f5ae2ffd640 1 -- 172.21.15.1:0/1334060774 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f5af415e770 con 0x7f5af40d1000 2024-03-24T10:36:01.344 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:36:01.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 msgr2=0x7f5ad406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 0x7f5ad406dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8036000 tx=0x7f5ae803d040 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 msgr2=0x7f5af41f12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c430 tx=0x7f5ae400c830 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 shutdown_connections 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f5ad406b810 0x7f5ad406dcd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5af415e390 0x7f5af41f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40d2bb0 0x7f5af415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 --2- 172.21.15.1:0/1334060774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af40d1000 0x7f5af41f12a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 >> 172.21.15.1:0/1334060774 conn(0x7f5af40d3ec0 msgr2=0x7f5af40cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 shutdown_connections 2024-03-24T10:36:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01.347+0000 7f5af89a0640 1 -- 172.21.15.1:0/1334060774 wait complete. 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:36:01.359 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:01.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.360 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:01.375 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42579sec preferred_lft 42579sec 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:36:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:01.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.378 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:36:01.440 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:36:01.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.441 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:36:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:36:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:36:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:36:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:36:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:36:01.500 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:36:01.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:36:01.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:01.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.635 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:36:01.651 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:36:01.651 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:36:01.651 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:01.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:36:01.717 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:36:01.718 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:01.718 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:01.772 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:01.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:01.773 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:01.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:01.894 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:01.954 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:01.954 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:01.957 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:01.957 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:01.957 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:01.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:01.957 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:02.009 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:36:02.009 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:36:02.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:02.078 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:02.135 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:02.135 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:02.135 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:02.135 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:02.136 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:02.137 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:02.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:02.138 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:02.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:36:02.137+0000 7f506fc3c480 -1 init, newargv = 0x557822282ab0 newargc=15 2024-03-24T10:36:02.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[57726]: starting ceph client 2024-03-24T10:36:02.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[57726]: starting fuse 2024-03-24T10:36:02.154 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:36:02.154 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:36:02.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:36:02.154 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:36:02.154 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:36:02.155 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:36:02.156 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:36:02.278 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.57726.asok 2024-03-24T10:36:02.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:02.288 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57726.asok 2024-03-24T10:36:02.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:02.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57726.asok status 2024-03-24T10:36:02.425 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:02.425 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:36:02.425 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:36:02.425 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "57726", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6156, 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6156 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:02.426 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 364669035 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 364669035 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6156 192.168.0.1:0/364669035", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/364669035", 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 25, 2024-03-24T10:36:02.427 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 50, 2024-03-24T10:36:02.428 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:36:02.428 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:36:02.428 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:36:02.428 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:02.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:02.435 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:36:02.435 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:02.450 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:36:02.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:02.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:02.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:02.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:02.518 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-24T10:36:02.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-24T10:36:02.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.730+0000 7f187d3bc640 1 Processor -- start 2024-03-24T10:36:02.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.730+0000 7f187d3bc640 1 -- start start 2024-03-24T10:36:02.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f187d3bc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 0x7f18780cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:02.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f187d3bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187806c930 con 0x7f1878144550 2024-03-24T10:36:02.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f187d3bc640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187806cb80 con 0x7f1878071cc0 2024-03-24T10:36:02.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f187d3bc640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f18780b4310 con 0x7f18780ccfe0 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f1876ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1878144550 0x7f18780b3520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40840/0 (socket says 172.21.15.1:40840) 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.731+0000 7f1876ffd640 1 -- 172.21.15.1:0/596363195 learned_addr learned my addr 172.21.15.1:0/596363195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f18767fc640 1 --2- 172.21.15.1:0/596363195 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 0x7f18780cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087537114 0 0) 0x7f187806c930 con 0x7f1878144550 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1860003520 con 0x7f1878144550 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415879006 0 0) 0x7f187806cb80 con 0x7f1878071cc0 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187806c930 con 0x7f1878071cc0 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f18767fc640 1 -- 172.21.15.1:0/596363195 >> v1:172.21.15.3:6789/0 conn(0x7f1878071cc0 legacy=0x7f187806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 805683788 0 0) 0x7f1860003520 con 0x7f1878144550 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f18767fc640 1 -- 172.21.15.1:0/596363195 >> v1:172.21.15.1:6789/0 conn(0x7f1878144550 legacy=0x7f18780b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:02.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.732+0000 7f18767fc640 1 -- 172.21.15.1:0/596363195 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187806ce60 con 0x7f18780ccfe0 2024-03-24T10:36:02.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.733+0000 7f18767fc640 1 --2- 172.21.15.1:0/596363195 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 0x7f18780cd440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f186c0097d0 tx=0x7f186c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf392703057ee2da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.733+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870001670 con 0x7f18780ccfe0 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.733+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186c02e050 con 0x7f18780ccfe0 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.733+0000 7f1875ffb640 1 -- 172.21.15.1:0/596363195 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c030710 con 0x7f18780ccfe0 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 -- 172.21.15.1:0/596363195 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 msgr2=0x7f18780cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 --2- 172.21.15.1:0/596363195 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 0x7f18780cd440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f186c0097d0 tx=0x7f186c02b800 comp rx=0 tx=0).stop 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 -- 172.21.15.1:0/596363195 shutdown_connections 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 --2- 172.21.15.1:0/596363195 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f18780ccfe0 0x7f18780cd440 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 -- 172.21.15.1:0/596363195 >> 172.21.15.1:0/596363195 conn(0x7f187806b4f0 msgr2=0x7f187806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 -- 172.21.15.1:0/596363195 shutdown_connections 2024-03-24T10:36:02.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.734+0000 7f187d3bc640 1 -- 172.21.15.1:0/596363195 wait complete. 2024-03-24T10:36:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.736+0000 7f187d3bc640 1 Processor -- start 2024-03-24T10:36:02.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.736+0000 7f187d3bc640 1 -- start start 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1878071cc0 0x7f1878159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1878144550 0x7f1878132ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f18781340d0 con 0x7f18780ccfe0 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f18781331e0 con 0x7f1878071cc0 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f187d3bc640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f1878133450 con 0x7f1878144550 2024-03-24T10:36:02.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54468/0 (socket says 172.21.15.1:54468) 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 -- 172.21.15.1:0/2749563417 learned_addr learned my addr 172.21.15.1:0/2749563417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f1876ffd640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1878071cc0 0x7f1878159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18777fe640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1878144550 0x7f1878132ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 -- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1878144550 msgr2=0x7f1878132ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1878144550 0x7f1878132ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 -- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1878071cc0 msgr2=0x7f1878159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1878071cc0 0x7f1878159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.737+0000 7f18767fc640 1 -- 172.21.15.1:0/2749563417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186c009400 con 0x7f18780ccfe0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f18767fc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f186c035040 tx=0x7f186c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c030710 con 0x7f18780ccfe0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186c030890 con 0x7f18780ccfe0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1878133730 con 0x7f18780ccfe0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c030c40 con 0x7f18780ccfe0 2024-03-24T10:36:02.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.738+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1878133cf0 con 0x7f18780ccfe0 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.739+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1878134210 con 0x7f18780ccfe0 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.739+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f186c030da0 con 0x7f18780ccfe0 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.740+0000 7f185f7fe640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 0x7f186406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.740+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f186c0abd50 con 0x7f18780ccfe0 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.740+0000 7f1876ffd640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 0x7f186406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:02.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.740+0000 7f1876ffd640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 0x7f186406dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1870000fd0 tx=0x7f1870001690 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:02.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.744+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1878134210 con 0x7f18780ccfe0 2024-03-24T10:36:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.995+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) -- 0x7f1878134210 con 0x7f18780ccfe0 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:02.997+0000 7f185f7fe640 1 -- 172.21.15.1:0/2749563417 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v18) ==== 183+0+140 (secure 0 0 0) 0x7f1878134210 con 0x7f18780ccfe0 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:36:02.997 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-24T10:36:02.998 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 msgr2=0x7f186406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 0x7f186406dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1870000fd0 tx=0x7f1870001690 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 msgr2=0x7f1878132760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f186c035040 tx=0x7f186c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 shutdown_connections 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f186406b810 0x7f186406dcd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f1878144550 0x7f1878132ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18780ccfe0 0x7f1878132760 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 --2- 172.21.15.1:0/2749563417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1878071cc0 0x7f1878159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 >> 172.21.15.1:0/2749563417 conn(0x7f187806b4f0 msgr2=0x7f187815b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 shutdown_connections 2024-03-24T10:36:03.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.000+0000 7f187d3bc640 1 -- 172.21.15.1:0/2749563417 wait complete. 2024-03-24T10:36:03.011 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:03.011 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:03.027 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.168+0000 7f519c15c640 1 Processor -- start 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.168+0000 7f519c15c640 1 -- start start 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519c15c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519c15c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519c15c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5194162b90 con 0x7f5194161610 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519c15c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f51940701e0 con 0x7f519406ee20 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519c15c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f51940b45a0 con 0x7f519406f7f0 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f51996f7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:54540/0 (socket says 172.21.15.1:54540) 2024-03-24T10:36:03.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f51996f7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:41438/0 (socket says 172.21.15.1:41438) 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f5199ef8640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5194161610 0x7f51940d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:59964/0 (socket says 172.21.15.1:59964) 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 -- 172.21.15.1:0/549784223 learned_addr learned my addr 172.21.15.1:0/549784223 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 -- 172.21.15.1:0/549784223 >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 msgr2=0x7f5194162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 --2- 172.21.15.1:0/549784223 >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 -- 172.21.15.1:0/549784223 >> v1:172.21.15.3:6789/0 conn(0x7f5194161610 legacy=0x7f51940d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f519a6f9640 1 -- 172.21.15.1:0/549784223 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51941619f0 con 0x7f519406f7f0 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.169+0000 7f51996f7640 1 --2- 172.21.15.1:0/549784223 >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.170+0000 7f519a6f9640 1 --2- 172.21.15.1:0/549784223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f518c009e50 tx=0x7f518c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35e46a8df68e31a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.170+0000 7f5198ef6640 1 -- 172.21.15.1:0/549784223 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c02cb00 con 0x7f519406f7f0 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.170+0000 7f5198ef6640 1 -- 172.21.15.1:0/549784223 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c02cc60 con 0x7f519406f7f0 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.170+0000 7f5198ef6640 1 -- 172.21.15.1:0/549784223 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0343f0 con 0x7f519406f7f0 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 -- 172.21.15.1:0/549784223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 msgr2=0x7f519406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 --2- 172.21.15.1:0/549784223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f518c009e50 tx=0x7f518c02bf60 comp rx=0 tx=0).stop 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 -- 172.21.15.1:0/549784223 shutdown_connections 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 --2- 172.21.15.1:0/549784223 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f519406f7f0 0x7f519406fca0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 --2- 172.21.15.1:0/549784223 >> v2:172.21.15.3:3300/0 conn(0x7f519406ee20 0x7f5194162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 -- 172.21.15.1:0/549784223 >> 172.21.15.1:0/549784223 conn(0x7f51940b3770 msgr2=0x7f51940b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 -- 172.21.15.1:0/549784223 shutdown_connections 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.171+0000 7f519c15c640 1 -- 172.21.15.1:0/549784223 wait complete. 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.172+0000 7f519c15c640 1 Processor -- start 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.172+0000 7f519c15c640 1 -- start start 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f519406f7f0 0x7f51940b7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5194161610 0x7f51940cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51941529b0 con 0x7f519406ee20 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f51940ca150 con 0x7f519406f7f0 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519c15c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f51940b79b0 con 0x7f5194161610 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f5199ef8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f5199ef8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54474/0 (socket says 172.21.15.1:54474) 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f5199ef8640 1 -- 172.21.15.1:0/862446145 learned_addr learned my addr 172.21.15.1:0/862446145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:03.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f519a6f9640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5194161610 0x7f51940cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f51996f7640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f519406f7f0 0x7f51940b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.173+0000 7f5199ef8640 1 -- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5194161610 msgr2=0x7f51940cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f5199ef8640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5194161610 0x7f51940cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f5199ef8640 1 -- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f519406f7f0 msgr2=0x7f51940b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f5199ef8640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f519406f7f0 0x7f51940b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f5199ef8640 1 -- 172.21.15.1:0/862446145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5184009400 con 0x7f519406ee20 2024-03-24T10:36:03.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f5199ef8640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f519000c400 tx=0x7f519000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900074e0 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5190007640 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c009b30 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190004a00 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.174+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51940ca640 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.175+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5194160e30 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.175+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5190004b60 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.176+0000 7f518a7fc640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 0x7f517806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.176+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f5190089920 con 0x7f519406ee20 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.176+0000 7f51996f7640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 0x7f517806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.176+0000 7f51996f7640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 0x7f517806dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f51940cf240 tx=0x7f5184009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.180+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f51900543f0 con 0x7f519406ee20 2024-03-24T10:36:03.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.402+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5194162b90 con 0x7f519406ee20 2024-03-24T10:36:03.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.403+0000 7f518a7fc640 1 -- 172.21.15.1:0/862446145 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f5194162b90 con 0x7f519406ee20 2024-03-24T10:36:03.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:03.404 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 msgr2=0x7f517806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 0x7f517806dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f51940cf240 tx=0x7f5184009190 comp rx=0 tx=0).stop 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 msgr2=0x7f51940cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f519000c400 tx=0x7f519000c8c0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 shutdown_connections 2024-03-24T10:36:03.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f517806b810 0x7f517806dcd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f5194161610 0x7f51940cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f519406f7f0 0x7f51940b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 --2- 172.21.15.1:0/862446145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519406ee20 0x7f51940cb120 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 >> 172.21.15.1:0/862446145 conn(0x7f51940b3770 msgr2=0x7f51940b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 shutdown_connections 2024-03-24T10:36:03.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.406+0000 7f519c15c640 1 -- 172.21.15.1:0/862446145 wait complete. 2024-03-24T10:36:03.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:03.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.523+0000 7f7c5ec42640 1 Processor -- start 2024-03-24T10:36:03.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.523+0000 7f7c5ec42640 1 -- start start 2024-03-24T10:36:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.523+0000 7f7c5ec42640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c580cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5ec42640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c5815c4c0 con 0x7f7c580b2e40 2024-03-24T10:36:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5ec42640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c580cd5a0 con 0x7f7c580cfd40 2024-03-24T10:36:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5ec42640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7c580cd880 con 0x7f7c5815c0e0 2024-03-24T10:36:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5d1df640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7c580b2e40 0x7f7c580ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40856/0 (socket says 172.21.15.1:40856) 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5d1df640 1 -- 172.21.15.1:0/1308002363 learned_addr learned my addr 172.21.15.1:0/1308002363 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5c9de640 1 --2- 172.21.15.1:0/1308002363 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c580cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 690008167 0 0) 0x7f7c5815c4c0 con 0x7f7c580b2e40 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c40003520 con 0x7f7c580b2e40 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302899517 0 0) 0x7f7c580cd5a0 con 0x7f7c580cfd40 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c5815c4c0 con 0x7f7c580cfd40 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5c9de640 1 -- 172.21.15.1:0/1308002363 >> v1:172.21.15.3:6789/0 conn(0x7f7c580cfd40 legacy=0x7f7c580ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5c9de640 1 -- 172.21.15.1:0/1308002363 >> v1:172.21.15.1:6789/0 conn(0x7f7c580b2e40 legacy=0x7f7c580ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.524+0000 7f7c5c9de640 1 -- 172.21.15.1:0/1308002363 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c580cdb60 con 0x7f7c5815c0e0 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.525+0000 7f7c5c9de640 1 --2- 172.21.15.1:0/1308002363 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c580cf800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009dc0 tx=0x7f7c5002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe0357d12312eba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.525+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5002ccd0 con 0x7f7c5815c0e0 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.525+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c5002ce30 con 0x7f7c5815c0e0 2024-03-24T10:36:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.525+0000 7f7c577fe640 1 -- 172.21.15.1:0/1308002363 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50034640 con 0x7f7c5815c0e0 2024-03-24T10:36:03.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 -- 172.21.15.1:0/1308002363 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 msgr2=0x7f7c580cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/1308002363 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c580cf800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009dc0 tx=0x7f7c5002bf20 comp rx=0 tx=0).stop 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 -- 172.21.15.1:0/1308002363 shutdown_connections 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/1308002363 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c580cf800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 -- 172.21.15.1:0/1308002363 >> 172.21.15.1:0/1308002363 conn(0x7f7c580d1c30 msgr2=0x7f7c580d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 -- 172.21.15.1:0/1308002363 shutdown_connections 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.526+0000 7f7c5ec42640 1 -- 172.21.15.1:0/1308002363 wait complete. 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.527+0000 7f7c5ec42640 1 Processor -- start 2024-03-24T10:36:03.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.527+0000 7f7c5ec42640 1 -- start start 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c580cfd40 0x7f7c58151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c5806f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7c580b7a80 con 0x7f7c580b2e40 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581f51f0 con 0x7f7c580cfd40 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5ec42640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7c581f54d0 con 0x7f7c5815c0e0 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54484/0 (socket says 172.21.15.1:54484) 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 -- 172.21.15.1:0/843634400 learned_addr learned my addr 172.21.15.1:0/843634400 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c57fff640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c580cfd40 0x7f7c58151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5d1df640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c5806f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 -- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 msgr2=0x7f7c5806f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c5806f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 -- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c580cfd40 msgr2=0x7f7c58151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c580cfd40 0x7f7c58151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.528+0000 7f7c5c9de640 1 -- 172.21.15.1:0/843634400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c50009a70 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c5c9de640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7c50037040 tx=0x7f7c50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50007000 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c5003d600 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c581f57b0 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5003d990 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.529+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581f5d70 con 0x7f7c580b2e40 2024-03-24T10:36:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.530+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c580cc380 con 0x7f7c580b2e40 2024-03-24T10:36:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.530+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7c5003b040 con 0x7f7c580b2e40 2024-03-24T10:36:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.531+0000 7f7c557fa640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 0x7f7c4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.531+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f7c500ac270 con 0x7f7c580b2e40 2024-03-24T10:36:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.531+0000 7f7c57fff640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 0x7f7c4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.531+0000 7f7c57fff640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 0x7f7c4406dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c007fd0 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.535+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c50076dd0 con 0x7f7c580b2e40 2024-03-24T10:36:03.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.760+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c581f5f60 con 0x7f7c580b2e40 2024-03-24T10:36:03.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.760+0000 7f7c557fa640 1 -- 172.21.15.1:0/843634400 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7f7c50009450 con 0x7f7c580b2e40 2024-03-24T10:36:03.763 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 msgr2=0x7f7c4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 0x7f7c4406dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c007fd0 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 msgr2=0x7f7c580b7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7c50037040 tx=0x7f7c50004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 shutdown_connections 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7c4406b810 0x7f7c4406dcd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7c5815c0e0 0x7f7c5806f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c580cfd40 0x7f7c58151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 --2- 172.21.15.1:0/843634400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c580b2e40 0x7f7c580b7540 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 >> 172.21.15.1:0/843634400 conn(0x7f7c580d1c30 msgr2=0x7f7c580ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 shutdown_connections 2024-03-24T10:36:03.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.762+0000 7f7c5ec42640 1 -- 172.21.15.1:0/843634400 wait complete. 2024-03-24T10:36:03.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.887+0000 7f703c047640 1 Processor -- start 2024-03-24T10:36:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.887+0000 7f703c047640 1 -- start start 2024-03-24T10:36:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f703c047640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 0x7f70340d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f703c047640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70340c7a40 con 0x7f70341615f0 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f703c047640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70340b55b0 con 0x7f7034162610 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f703c047640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f70340cfd50 con 0x7f70340b4e40 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7039de3640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f70341615f0 0x7f7034161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40864/0 (socket says 172.21.15.1:40864) 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7039de3640 1 -- 172.21.15.1:0/3215057867 learned_addr learned my addr 172.21.15.1:0/3215057867 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f70395e2640 1 --2- 172.21.15.1:0/3215057867 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 0x7f70340d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106308034 0 0) 0x7f70340c7a40 con 0x7f70341615f0 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7020003520 con 0x7f70341615f0 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3663721466 0 0) 0x7f70340b55b0 con 0x7f7034162610 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70340c7a40 con 0x7f7034162610 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f70395e2640 1 -- 172.21.15.1:0/3215057867 >> v1:172.21.15.3:6789/0 conn(0x7f7034162610 legacy=0x7f70340d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f70395e2640 1 -- 172.21.15.1:0/3215057867 >> v1:172.21.15.1:6789/0 conn(0x7f70341615f0 legacy=0x7f7034161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1678453434 0 0) 0x7f7020003520 con 0x7f70341615f0 2024-03-24T10:36:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.888+0000 7f70395e2640 1 -- 172.21.15.1:0/3215057867 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70340b52d0 con 0x7f70340b4e40 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.889+0000 7f70395e2640 1 --2- 172.21.15.1:0/3215057867 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 0x7f70340d4a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f70240097d0 tx=0x7f702402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59694ea23baf44c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.889+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030001670 con 0x7f70340b4e40 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.889+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702402e050 con 0x7f70340b4e40 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.889+0000 7f7038de1640 1 -- 172.21.15.1:0/3215057867 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024030730 con 0x7f70340b4e40 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 -- 172.21.15.1:0/3215057867 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 msgr2=0x7f70340d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 --2- 172.21.15.1:0/3215057867 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 0x7f70340d4a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f70240097d0 tx=0x7f702402b800 comp rx=0 tx=0).stop 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 -- 172.21.15.1:0/3215057867 shutdown_connections 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 --2- 172.21.15.1:0/3215057867 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f70340b4e40 0x7f70340d4a10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 -- 172.21.15.1:0/3215057867 >> 172.21.15.1:0/3215057867 conn(0x7f70340d3ef0 msgr2=0x7f70340d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 -- 172.21.15.1:0/3215057867 shutdown_connections 2024-03-24T10:36:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.890+0000 7f703c047640 1 -- 172.21.15.1:0/3215057867 wait complete. 2024-03-24T10:36:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.891+0000 7f703c047640 1 Processor -- start 2024-03-24T10:36:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.891+0000 7f703c047640 1 -- start start 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7034162610 0x7f7034156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 0x7f70340c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f70340b7bb0 con 0x7f70341615f0 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f70340c9ce0 con 0x7f70340c92f0 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703c047640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f70341556a0 con 0x7f7034162610 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54500/0 (socket says 172.21.15.1:54500) 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 -- 172.21.15.1:0/388638780 learned_addr learned my addr 172.21.15.1:0/388638780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f70395e2640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7034162610 0x7f7034156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703a5e4640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 0x7f70340c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 -- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7034162610 msgr2=0x7f7034156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7034162610 0x7f7034156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 -- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 msgr2=0x7f70340c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 0x7f70340c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f7039de3640 1 -- 172.21.15.1:0/388638780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7024009400 con 0x7f70341615f0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.892+0000 7f703a5e4640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 0x7f70340c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f7039de3640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f703000c780 tx=0x7f703000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030005ad0 con 0x7f70341615f0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7030005c30 con 0x7f70341615f0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70341f75a0 con 0x7f70341615f0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030007070 con 0x7f70341615f0 2024-03-24T10:36:03.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.893+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70340b7210 con 0x7f70341615f0 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.894+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703404f470 con 0x7f70341615f0 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.894+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f70300071d0 con 0x7f70341615f0 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.895+0000 7f702a7fc640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 0x7f701806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.895+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f70300897e0 con 0x7f70341615f0 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.895+0000 7f70395e2640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 0x7f701806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.895+0000 7f70395e2640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 0x7f701806dd00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7024034000 tx=0x7f7024008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:03.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:03.899+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f70300542c0 con 0x7f70341615f0 2024-03-24T10:36:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.081+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f703408d640 con 0x7f70341615f0 2024-03-24T10:36:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.082+0000 7f702a7fc640 1 -- 172.21.15.1:0/388638780 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7f7030057980 con 0x7f70341615f0 2024-03-24T10:36:04.082 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 msgr2=0x7f701806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 0x7f701806dd00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7024034000 tx=0x7f7024008030 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 msgr2=0x7f70341f7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f703000c780 tx=0x7f703000cc40 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 shutdown_connections 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f701806b840 0x7f701806dd00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70340c92f0 0x7f70340c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7034162610 0x7f7034156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 --2- 172.21.15.1:0/388638780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70341615f0 0x7f70341f7060 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 >> 172.21.15.1:0/388638780 conn(0x7f70340d3ef0 msgr2=0x7f70340d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 shutdown_connections 2024-03-24T10:36:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.084+0000 7f703c047640 1 -- 172.21.15.1:0/388638780 wait complete. 2024-03-24T10:36:04.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.202+0000 7f8a2810c640 1 Processor -- start 2024-03-24T10:36:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.202+0000 7f8a2810c640 1 -- start start 2024-03-24T10:36:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a2810c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 0x7f8a200cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a2810c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 0x7f8a200cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a2810c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a200b0b70 con 0x7f8a2015e390 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a2810c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f8a200d1d60 con 0x7f8a200d2a20 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a2810c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8a200b0de0 con 0x7f8a200b2310 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a25ea8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a2015e390 0x7f8a200d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40878/0 (socket says 172.21.15.1:40878) 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a25ea8640 1 -- 172.21.15.1:0/2320207643 learned_addr learned my addr 172.21.15.1:0/2320207643 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a256a7640 1 --2- 172.21.15.1:0/2320207643 >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 0x7f8a200cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a266a9640 1 --2- 172.21.15.1:0/2320207643 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 0x7f8a200cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989830304 0 0) 0x7f8a200b0b70 con 0x7f8a2015e390 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.203+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a0c003520 con 0x7f8a2015e390 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 599664443 0 0) 0x7f8a0c003520 con 0x7f8a2015e390 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a266a9640 1 -- 172.21.15.1:0/2320207643 >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 msgr2=0x7f8a200cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a266a9640 1 --2- 172.21.15.1:0/2320207643 >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 0x7f8a200cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a266a9640 1 -- 172.21.15.1:0/2320207643 >> v1:172.21.15.1:6789/0 conn(0x7f8a2015e390 legacy=0x7f8a200d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a266a9640 1 -- 172.21.15.1:0/2320207643 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2015e770 con 0x7f8a200b2310 2024-03-24T10:36:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.204+0000 7f8a256a7640 1 --2- 172.21.15.1:0/2320207643 >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 0x7f8a200cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a266a9640 1 --2- 172.21.15.1:0/2320207643 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 0x7f8a200cf010 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a1802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1da7d1f57325f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c001670 con 0x7f8a200b2310 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1802c8a0 con 0x7f8a200b2310 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a24ea6640 1 -- 172.21.15.1:0/2320207643 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1802cc10 con 0x7f8a200b2310 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 -- 172.21.15.1:0/2320207643 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 msgr2=0x7f8a200cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 --2- 172.21.15.1:0/2320207643 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 0x7f8a200cf010 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a1802bd70 comp rx=0 tx=0).stop 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 -- 172.21.15.1:0/2320207643 shutdown_connections 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 --2- 172.21.15.1:0/2320207643 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200b2310 0x7f8a200cf010 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 --2- 172.21.15.1:0/2320207643 >> v2:172.21.15.3:3300/0 conn(0x7f8a200d2a20 0x7f8a200cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 -- 172.21.15.1:0/2320207643 >> 172.21.15.1:0/2320207643 conn(0x7f8a200d3ee0 msgr2=0x7f8a200d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 -- 172.21.15.1:0/2320207643 shutdown_connections 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.205+0000 7f8a2810c640 1 -- 172.21.15.1:0/2320207643 wait complete. 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 Processor -- start 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 -- start start 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200d2a20 0x7f8a2015f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2015e390 0x7f8a200c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200b8b90 con 0x7f8a200b2310 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200b94f0 con 0x7f8a2015e390 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a2810c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2015f870 con 0x7f8a200d2a20 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a25ea8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a25ea8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54508/0 (socket says 172.21.15.1:54508) 2024-03-24T10:36:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.207+0000 7f8a25ea8640 1 -- 172.21.15.1:0/1979062235 learned_addr learned my addr 172.21.15.1:0/1979062235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a266a9640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2015e390 0x7f8a200c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a256a7640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200d2a20 0x7f8a2015f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 -- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200d2a20 msgr2=0x7f8a2015f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200d2a20 0x7f8a2015f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 -- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2015e390 msgr2=0x7f8a200c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2015e390 0x7f8a200c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 -- 172.21.15.1:0/1979062235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a10009400 con 0x7f8a200b2310 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a25ea8640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00c750 tx=0x7f8a1c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c005ad0 con 0x7f8a200b2310 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c005c30 con 0x7f8a200b2310 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a18009a70 con 0x7f8a200b2310 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c007070 con 0x7f8a200b2310 2024-03-24T10:36:04.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.208+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a200b84d0 con 0x7f8a200b2310 2024-03-24T10:36:04.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.209+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a20073470 con 0x7f8a200b2310 2024-03-24T10:36:04.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.210+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8a1c0071d0 con 0x7f8a200b2310 2024-03-24T10:36:04.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.210+0000 7f8a167fc640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 0x7f8a0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.210+0000 7f8a256a7640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 0x7f8a0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.210+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f8a1c089f00 con 0x7f8a200b2310 2024-03-24T10:36:04.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.211+0000 7f8a256a7640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 0x7f8a0406dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a200d2f10 tx=0x7f8a10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.215+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8a1c054930 con 0x7f8a200b2310 2024-03-24T10:36:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.529+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a2004f4e0 con 0x7f8a200b2310 2024-03-24T10:36:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.529+0000 7f8a167fc640 1 -- 172.21.15.1:0/1979062235 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7f8a1c075e60 con 0x7f8a200b2310 2024-03-24T10:36:04.530 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 msgr2=0x7f8a0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 0x7f8a0406dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a200d2f10 tx=0x7f8a10008040 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 msgr2=0x7f8a200b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00c750 tx=0x7f8a1c00cc10 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 shutdown_connections 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a0406b810 0x7f8a0406dcd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2015e390 0x7f8a200c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a200d2a20 0x7f8a2015f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 --2- 172.21.15.1:0/1979062235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a200b2310 0x7f8a200b8fb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 >> 172.21.15.1:0/1979062235 conn(0x7f8a200d3ee0 msgr2=0x7f8a200cf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 shutdown_connections 2024-03-24T10:36:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.531+0000 7f8a2810c640 1 -- 172.21.15.1:0/1979062235 wait complete. 2024-03-24T10:36:04.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:36:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.645+0000 7f8a32d0f640 1 Processor -- start 2024-03-24T10:36:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.645+0000 7f8a32d0f640 1 -- start start 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a32d0f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a2c15e390 0x7f8a2c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a32d0f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 0x7f8a2c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a32d0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2c15e770 con 0x7f8a2c0b3320 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a32d0f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2c0b3190 con 0x7f8a2c15e390 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a32d0f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2c0d4e40 con 0x7f8a2c0d1000 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a30aab640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a2c15e390 0x7f8a2c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a312ac640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a2c0b3320 0x7f8a2c0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40888/0 (socket says 172.21.15.1:40888) 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a312ac640 1 -- 172.21.15.1:0/1004995549 learned_addr learned my addr 172.21.15.1:0/1004995549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.646+0000 7f8a2bfff640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 0x7f8a2c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.647+0000 7f8a30aab640 1 -- 172.21.15.1:0/1004995549 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 msgr2=0x7f8a2c0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.647+0000 7f8a30aab640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 0x7f8a2c0d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.647+0000 7f8a30aab640 1 -- 172.21.15.1:0/1004995549 >> v1:172.21.15.1:6789/0 conn(0x7f8a2c0b3320 legacy=0x7f8a2c0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.647+0000 7f8a30aab640 1 -- 172.21.15.1:0/1004995549 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c0d1e40 con 0x7f8a2c15e390 2024-03-24T10:36:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.647+0000 7f8a2bfff640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 0x7f8a2c0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.648+0000 7f8a30aab640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c0d24a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8a200097a0 tx=0x7f8a2002b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a1f1ed079dfa84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.648+0000 7f8a2b7fe640 1 -- 172.21.15.1:0/1004995549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20030710 con 0x7f8a2c15e390 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.648+0000 7f8a2b7fe640 1 -- 172.21.15.1:0/1004995549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2002e050 con 0x7f8a2c15e390 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.648+0000 7f8a2b7fe640 1 -- 172.21.15.1:0/1004995549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20030a90 con 0x7f8a2c15e390 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 -- 172.21.15.1:0/1004995549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 msgr2=0x7f8a2c0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c0d24a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8a200097a0 tx=0x7f8a2002b5d0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 -- 172.21.15.1:0/1004995549 shutdown_connections 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0d1000 0x7f8a2c0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/1004995549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c0d24a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 -- 172.21.15.1:0/1004995549 >> 172.21.15.1:0/1004995549 conn(0x7f8a2c0d3ec0 msgr2=0x7f8a2c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 -- 172.21.15.1:0/1004995549 shutdown_connections 2024-03-24T10:36:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.649+0000 7f8a32d0f640 1 -- 172.21.15.1:0/1004995549 wait complete. 2024-03-24T10:36:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.650+0000 7f8a32d0f640 1 Processor -- start 2024-03-24T10:36:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.650+0000 7f8a32d0f640 1 -- start start 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 0x7f8a2c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 0x7f8a2c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c1576a0 con 0x7f8a2c15e390 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c0b7910 con 0x7f8a2c0d1000 2024-03-24T10:36:04.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a32d0f640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2c15ad70 con 0x7f8a2c0b3320 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54526/0 (socket says 172.21.15.1:54526) 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 -- 172.21.15.1:0/108804595 learned_addr learned my addr 172.21.15.1:0/108804595 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a30aab640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 0x7f8a2c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a2bfff640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 0x7f8a2c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 -- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 msgr2=0x7f8a2c157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 0x7f8a2c157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 -- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 msgr2=0x7f8a2c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 0x7f8a2c15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 -- 172.21.15.1:0/108804595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a20009400 con 0x7f8a2c15e390 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a30aab640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 0x7f8a2c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a2bfff640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 0x7f8a2c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a312ac640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400caf0 tx=0x7f8a2400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.651+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2400a890 con 0x7f8a2c15e390 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.652+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2400a9f0 con 0x7f8a2c15e390 2024-03-24T10:36:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.652+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2c15b010 con 0x7f8a2c15e390 2024-03-24T10:36:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.652+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2400ade0 con 0x7f8a2c15e390 2024-03-24T10:36:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.652+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2c15b2f0 con 0x7f8a2c15e390 2024-03-24T10:36:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.652+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2c04f4e0 con 0x7f8a2c15e390 2024-03-24T10:36:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.653+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8a240070d0 con 0x7f8a2c15e390 2024-03-24T10:36:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.654+0000 7f8a297fa640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 0x7f8a1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.654+0000 7f8a30aab640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 0x7f8a1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.654+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f8a24089b80 con 0x7f8a2c15e390 2024-03-24T10:36:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.654+0000 7f8a30aab640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 0x7f8a1006dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a20034000 tx=0x7f8a20008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.658+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8a240545a0 con 0x7f8a2c15e390 2024-03-24T10:36:04.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.852+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8a2c15e770 con 0x7f8a2c15e390 2024-03-24T10:36:04.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.852+0000 7f8a297fa640 1 -- 172.21.15.1:0/108804595 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) ==== 80+0+0 (secure 0 0 0) 0x7f8a2c15e770 con 0x7f8a2c15e390 2024-03-24T10:36:04.853 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.854+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 msgr2=0x7f8a1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.854+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 0x7f8a1006dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a20034000 tx=0x7f8a20008030 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.854+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 msgr2=0x7f8a2c15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.854+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400caf0 tx=0x7f8a2400a060 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.854+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 shutdown_connections 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f8a1006b810 0x7f8a1006dcd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a2c15e390 0x7f8a2c15bb20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c0d1000 0x7f8a2c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 --2- 172.21.15.1:0/108804595 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f8a2c0b3320 0x7f8a2c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 >> 172.21.15.1:0/108804595 conn(0x7f8a2c0d3ec0 msgr2=0x7f8a2c0d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 shutdown_connections 2024-03-24T10:36:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:04.855+0000 7f8a32d0f640 1 -- 172.21.15.1:0/108804595 wait complete. 2024-03-24T10:36:04.866 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:04.866 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:04.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:05.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.029+0000 7f3273f43640 1 Processor -- start 2024-03-24T10:36:05.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.029+0000 7f3273f43640 1 -- start start 2024-03-24T10:36:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f3273f43640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f326c0b31b0 0x7f326c070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f3273f43640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c0cdd60 con 0x7f326c0d00f0 2024-03-24T10:36:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f3273f43640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c06e200 con 0x7f326c144770 2024-03-24T10:36:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f3273f43640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f326c06eaa0 con 0x7f326c0b31b0 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32724e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f326c0b31b0 0x7f326c070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32714de640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f326c0d00f0 0x7f326c0b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40898/0 (socket says 172.21.15.1:40898) 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32714de640 1 -- 172.21.15.1:0/2114515766 learned_addr learned my addr 172.21.15.1:0/2114515766 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32724e0640 1 -- 172.21.15.1:0/2114515766 >> v1:172.21.15.3:6789/0 conn(0x7f326c144770 legacy=0x7f326c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32724e0640 1 -- 172.21.15.1:0/2114515766 >> v1:172.21.15.1:6789/0 conn(0x7f326c0d00f0 legacy=0x7f326c0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f3270cdd640 1 -- 172.21.15.1:0/2114515766 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078892320 0 0) 0x7f326c0cdd60 con 0x7f326c0d00f0 2024-03-24T10:36:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.030+0000 7f32724e0640 1 -- 172.21.15.1:0/2114515766 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c06e4e0 con 0x7f326c0b31b0 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.031+0000 7f32724e0640 1 --2- 172.21.15.1:0/2114515766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c0b31b0 0x7f326c070b90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5cd4f031cf1e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.031+0000 7f3270cdd640 1 -- 172.21.15.1:0/2114515766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326402cd00 con 0x7f326c0b31b0 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.031+0000 7f3270cdd640 1 -- 172.21.15.1:0/2114515766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326402ce60 con 0x7f326c0b31b0 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.031+0000 7f3270cdd640 1 -- 172.21.15.1:0/2114515766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264034660 con 0x7f326c0b31b0 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 -- 172.21.15.1:0/2114515766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c0b31b0 msgr2=0x7f326c070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 --2- 172.21.15.1:0/2114515766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c0b31b0 0x7f326c070b90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f326402bf50 comp rx=0 tx=0).stop 2024-03-24T10:36:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 -- 172.21.15.1:0/2114515766 shutdown_connections 2024-03-24T10:36:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 --2- 172.21.15.1:0/2114515766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c0b31b0 0x7f326c070b90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 -- 172.21.15.1:0/2114515766 >> 172.21.15.1:0/2114515766 conn(0x7f326c0b17a0 msgr2=0x7f326c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 -- 172.21.15.1:0/2114515766 shutdown_connections 2024-03-24T10:36:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.032+0000 7f3273f43640 1 -- 172.21.15.1:0/2114515766 wait complete. 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.033+0000 7f3273f43640 1 Processor -- start 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.033+0000 7f3273f43640 1 -- start start 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f326c0b31b0 0x7f326c0ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0d00f0 0x7f326c1f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f326c0aeab0 con 0x7f326c144770 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f326c0aed90 con 0x7f326c0d00f0 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f326c0af070 con 0x7f326c0b31b0 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54548/0 (socket says 172.21.15.1:54548) 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 -- 172.21.15.1:0/2359771565 learned_addr learned my addr 172.21.15.1:0/2359771565 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32714de640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0d00f0 0x7f326c1f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3271cdf640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f326c0b31b0 0x7f326c0ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 -- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f326c0b31b0 msgr2=0x7f326c0ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f326c0b31b0 0x7f326c0ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 -- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0d00f0 msgr2=0x7f326c1f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0d00f0 0x7f326c1f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 -- 172.21.15.1:0/2359771565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3264009a70 con 0x7f326c144770 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32724e0640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f3264004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326403d600 con 0x7f326c144770 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326403d780 con 0x7f326c144770 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326c15c020 con 0x7f326c144770 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.034+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326403db30 con 0x7f326c144770 2024-03-24T10:36:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.035+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326c1f6fa0 con 0x7f326c144770 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.036+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326c15be80 con 0x7f326c144770 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.036+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f326403b040 con 0x7f326c144770 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.036+0000 7f32627fc640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 0x7f325006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.037+0000 7f3271cdf640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 0x7f325006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.037+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f32640ac200 con 0x7f326c144770 2024-03-24T10:36:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.037+0000 7f3271cdf640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 0x7f325006dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3268000f60 tx=0x7f3268009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.041+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f326c15be80 con 0x7f326c144770 2024-03-24T10:36:05.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.222+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f326c15be80 con 0x7f326c144770 2024-03-24T10:36:05.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.223+0000 7f32627fc640 1 -- 172.21.15.1:0/2359771565 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f326c15be80 con 0x7f326c144770 2024-03-24T10:36:05.223 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:05.223 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.224+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 msgr2=0x7f325006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.224+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 0x7f325006dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3268000f60 tx=0x7f3268009040 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.224+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 msgr2=0x7f326c139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.224+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3264009d90 tx=0x7f3264004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 shutdown_connections 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f325006b810 0x7f325006dcd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326c144770 0x7f326c139970 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0d00f0 0x7f326c1f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 --2- 172.21.15.1:0/2359771565 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f326c0b31b0 0x7f326c0ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 >> 172.21.15.1:0/2359771565 conn(0x7f326c0b17a0 msgr2=0x7f326c0b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 shutdown_connections 2024-03-24T10:36:05.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.225+0000 7f3273f43640 1 -- 172.21.15.1:0/2359771565 wait complete. 2024-03-24T10:36:05.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:05.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 Processor -- start 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 -- start start 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee9c145730 0x7fee9c15f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee9c06d120 0x7fee9c15ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee9c145b10 con 0x7fee9c070bd0 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fee9c06d620 con 0x7fee9c145730 2024-03-24T10:36:05.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.341+0000 7feea189d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fee9c06d900 con 0x7fee9c06d120 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee9c145730 0x7fee9c15f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fee9c070bd0 0x7fee9c071000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40902/0 (socket says 172.21.15.1:40902) 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9b7fe640 1 -- 172.21.15.1:0/4143648833 learned_addr learned my addr 172.21.15.1:0/4143648833 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee93fff640 1 --2- 172.21.15.1:0/4143648833 >> v2:172.21.15.3:3300/0 conn(0x7fee9c06d120 0x7fee9c15ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9affd640 1 -- 172.21.15.1:0/4143648833 >> v2:172.21.15.3:3300/0 conn(0x7fee9c06d120 msgr2=0x7fee9c15ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9affd640 1 --2- 172.21.15.1:0/4143648833 >> v2:172.21.15.3:3300/0 conn(0x7fee9c06d120 0x7fee9c15ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9affd640 1 -- 172.21.15.1:0/4143648833 >> v1:172.21.15.1:6789/0 conn(0x7fee9c070bd0 legacy=0x7fee9c071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.342+0000 7fee9affd640 1 -- 172.21.15.1:0/4143648833 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c06dbe0 con 0x7fee9c145730 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7fee9affd640 1 --2- 172.21.15.1:0/4143648833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c145730 0x7fee9c15f9d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fee9c06d050 tx=0x7fee8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fd39691b971fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7fee9a7fc640 1 -- 172.21.15.1:0/4143648833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c031830 con 0x7fee9c145730 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7fee9a7fc640 1 -- 172.21.15.1:0/4143648833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8c031990 con 0x7fee9c145730 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7fee9a7fc640 1 -- 172.21.15.1:0/4143648833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c031d20 con 0x7fee9c145730 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 -- 172.21.15.1:0/4143648833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c145730 msgr2=0x7fee9c15f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 --2- 172.21.15.1:0/4143648833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c145730 0x7fee9c15f9d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fee9c06d050 tx=0x7fee8c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 -- 172.21.15.1:0/4143648833 shutdown_connections 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 --2- 172.21.15.1:0/4143648833 >> v2:172.21.15.3:3300/0 conn(0x7fee9c06d120 0x7fee9c15ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 --2- 172.21.15.1:0/4143648833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c145730 0x7fee9c15f9d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.343+0000 7feea189d640 1 -- 172.21.15.1:0/4143648833 >> 172.21.15.1:0/4143648833 conn(0x7fee9c0b1710 msgr2=0x7fee9c0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.344+0000 7feea189d640 1 -- 172.21.15.1:0/4143648833 shutdown_connections 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.344+0000 7feea189d640 1 -- 172.21.15.1:0/4143648833 wait complete. 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 Processor -- start 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 -- start start 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 0x7fee9c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee9c145730 0x7fee9c1f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1f2890 con 0x7fee9c06d120 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1f1ec0 con 0x7fee9c070bd0 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.345+0000 7feea189d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fee9c1f1890 con 0x7fee9c145730 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54568/0 (socket says 172.21.15.1:54568) 2024-03-24T10:36:05.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 -- 172.21.15.1:0/3751172898 learned_addr learned my addr 172.21.15.1:0/3751172898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee93fff640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 0x7fee9c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9b7fe640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee9c145730 0x7fee9c1f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 -- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee9c145730 msgr2=0x7fee9c1f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee9c145730 0x7fee9c1f4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 -- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 msgr2=0x7fee9c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 0x7fee9c0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 -- 172.21.15.1:0/3751172898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c009400 con 0x7fee9c06d120 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee93fff640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 0x7fee9c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:05.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee9affd640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c03aa60 con 0x7fee9c06d120 2024-03-24T10:36:05.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8c03abe0 con 0x7fee9c06d120 2024-03-24T10:36:05.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9c1f5900 con 0x7fee9c06d120 2024-03-24T10:36:05.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c0443f0 con 0x7fee9c06d120 2024-03-24T10:36:05.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.346+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c1f1b50 con 0x7fee9c06d120 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.347+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9c06e8e0 con 0x7fee9c06d120 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.348+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee8c04d450 con 0x7fee9c06d120 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.348+0000 7fee98ff9640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 0x7fee7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.348+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fee8c0ac3a0 con 0x7fee9c06d120 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.348+0000 7fee93fff640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 0x7fee7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.349+0000 7fee93fff640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 0x7fee7c06dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fee9c13c610 tx=0x7fee84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.357+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee8c076f70 con 0x7fee9c06d120 2024-03-24T10:36:05.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.536+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee9c04f4e0 con 0x7fee9c06d120 2024-03-24T10:36:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.537+0000 7fee98ff9640 1 -- 172.21.15.1:0/3751172898 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7fee8c035020 con 0x7fee9c06d120 2024-03-24T10:36:05.537 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:05.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 msgr2=0x7fee7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 0x7fee7c06dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fee9c13c610 tx=0x7fee84008040 comp rx=0 tx=0).stop 2024-03-24T10:36:05.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 msgr2=0x7fee9c1f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 shutdown_connections 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fee7c06b810 0x7fee7c06dcd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fee9c145730 0x7fee9c1f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee9c070bd0 0x7fee9c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 --2- 172.21.15.1:0/3751172898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee9c06d120 0x7fee9c1f2350 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 >> 172.21.15.1:0/3751172898 conn(0x7fee9c0b1710 msgr2=0x7fee9c144a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 shutdown_connections 2024-03-24T10:36:05.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.539+0000 7feea189d640 1 -- 172.21.15.1:0/3751172898 wait complete. 2024-03-24T10:36:05.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:05.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.655+0000 7efefc7ac640 1 Processor -- start 2024-03-24T10:36:05.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.655+0000 7efefc7ac640 1 -- start start 2024-03-24T10:36:05.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefc7ac640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efef41615f0 0x7efef40d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefc7ac640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40c7a40 con 0x7efef4162610 2024-03-24T10:36:05.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefc7ac640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40b55b0 con 0x7efef40b4e40 2024-03-24T10:36:05.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefc7ac640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7efef40cfd50 con 0x7efef41615f0 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefad49640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efef4162610 0x7efef40d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40908/0 (socket says 172.21.15.1:40908) 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.656+0000 7efefad49640 1 -- 172.21.15.1:0/3342813899 learned_addr learned my addr 172.21.15.1:0/3342813899 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefa548640 1 --2- 172.21.15.1:0/3342813899 >> v2:172.21.15.3:3300/0 conn(0x7efef41615f0 0x7efef40d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734099838 0 0) 0x7efef40c7a40 con 0x7efef4162610 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee0003520 con 0x7efef4162610 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3416078850 0 0) 0x7efee0003520 con 0x7efef4162610 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 >> v2:172.21.15.3:3300/0 conn(0x7efef41615f0 msgr2=0x7efef40d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 --2- 172.21.15.1:0/3342813899 >> v2:172.21.15.3:3300/0 conn(0x7efef41615f0 0x7efef40d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 >> v1:172.21.15.3:6789/0 conn(0x7efef40b4e40 legacy=0x7efef4161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40b52d0 con 0x7efef4162610 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7efeec0019f0 con 0x7efef4162610 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 >> v1:172.21.15.1:6789/0 conn(0x7efef4162610 legacy=0x7efef40d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0003970 0x7efee0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efee00047c0 0x7efee0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 0x7efee000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efef40cfd50 con 0x7efee0005180 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efee000c020 con 0x7efee00047c0 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efee000c1e0 con 0x7efee0003970 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 0x7efee000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefa548640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0003970 0x7efee0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efef9d47640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efee00047c0 0x7efee0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 -- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0003970 msgr2=0x7efee0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0003970 0x7efee0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 -- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efee00047c0 msgr2=0x7efee0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efee00047c0 0x7efee0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.657+0000 7efefad49640 1 -- 172.21.15.1:0/3342813899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee0003520 con 0x7efee0005180 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.658+0000 7efefad49640 1 -- 172.21.15.1:0/3342813899 reap_dead start 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefad49640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 0x7efee000b980 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a830 tx=0x7efee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234082e1b39fe124 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec0072d0 con 0x7efee0005180 2024-03-24T10:36:05.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec02f070 con 0x7efee0005180 2024-03-24T10:36:05.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efef9546640 1 -- 172.21.15.1:0/3342813899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec0076b0 con 0x7efee0005180 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 -- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 msgr2=0x7efee000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 0x7efee000b980 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a830 tx=0x7efee00057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 -- 172.21.15.1:0/3342813899 shutdown_connections 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3342813899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efee0005180 0x7efee000b980 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 -- 172.21.15.1:0/3342813899 >> 172.21.15.1:0/3342813899 conn(0x7efef40d3ef0 msgr2=0x7efef40d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 -- 172.21.15.1:0/3342813899 shutdown_connections 2024-03-24T10:36:05.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.659+0000 7efefc7ac640 1 -- 172.21.15.1:0/3342813899 wait complete. 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.660+0000 7efefc7ac640 1 Processor -- start 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 -- start start 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 0x7efef40b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 0x7efef4155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efef4155870 con 0x7efef40b7220 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7efef40b8820 con 0x7efef4158c90 2024-03-24T10:36:05.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefc7ac640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7efef40b8b00 con 0x7efee0005180 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefad49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefad49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54586/0 (socket says 172.21.15.1:54586) 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.661+0000 7efefad49640 1 -- 172.21.15.1:0/3989703426 learned_addr learned my addr 172.21.15.1:0/3989703426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefa548640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 0x7efef40b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efef9d47640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 0x7efef4155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 -- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 msgr2=0x7efef40b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 0x7efef40b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 -- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 msgr2=0x7efef4155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 0x7efef4155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 -- 172.21.15.1:0/3989703426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40b8de0 con 0x7efef40b7220 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefa548640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 0x7efef40b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efef9d47640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 0x7efef4155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefad49640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7efeec001580 tx=0x7efeec001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec004e20 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec0017b0 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec001b20 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef40b6a20 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.662+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef41590d0 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.663+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef40b6f40 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.663+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efeec031030 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.664+0000 7efeeaffd640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 0x7efed806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.664+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7efeec0abed0 con 0x7efef40b7220 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.664+0000 7efefa548640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 0x7efed806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.664+0000 7efefa548640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 0x7efed806dd00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efef0009a40 tx=0x7efef0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.672+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efef40b6f40 con 0x7efef40b7220 2024-03-24T10:36:05.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.920+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efef40b6f40 con 0x7efef40b7220 2024-03-24T10:36:05.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.921+0000 7efeeaffd640 1 -- 172.21.15.1:0/3989703426 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7efef40b6f40 con 0x7efef40b7220 2024-03-24T10:36:05.921 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:05.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 msgr2=0x7efed806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 0x7efed806dd00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efef0009a40 tx=0x7efef0008040 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 msgr2=0x7efef40c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7efeec001580 tx=0x7efeec001d80 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 shutdown_connections 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7efed806b840 0x7efed806dd00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef40b7220 0x7efef40c9450 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef4158c90 0x7efef4155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 --2- 172.21.15.1:0/3989703426 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7efee0005180 0x7efef40b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 >> 172.21.15.1:0/3989703426 conn(0x7efef40d3ef0 msgr2=0x7efef406be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 shutdown_connections 2024-03-24T10:36:05.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:05.923+0000 7efefc7ac640 1 -- 172.21.15.1:0/3989703426 wait complete. 2024-03-24T10:36:05.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.038+0000 7ff8c12ae640 1 Processor -- start 2024-03-24T10:36:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.038+0000 7ff8c12ae640 1 -- start start 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8c12ae640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8c12ae640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8bc0b2310 0x7ff8bc0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8c12ae640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8bc0b0b70 con 0x7ff8bc15e390 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8c12ae640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff8bc0d1d60 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8c12ae640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7ff8bc0b0de0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54592/0 (socket says 172.21.15.1:54592) 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 -- 172.21.15.1:0/3683899900 learned_addr learned my addr 172.21.15.1:0/3683899900 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8baffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff8bc15e390 0x7ff8bc0d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40910/0 (socket says 172.21.15.1:40910) 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8bb7fe640 1 --2- 172.21.15.1:0/3683899900 >> v2:172.21.15.3:3300/0 conn(0x7ff8bc0b2310 0x7ff8bc0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 -- 172.21.15.1:0/3683899900 >> v2:172.21.15.3:3300/0 conn(0x7ff8bc0b2310 msgr2=0x7ff8bc0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/3683899900 >> v2:172.21.15.3:3300/0 conn(0x7ff8bc0b2310 0x7ff8bc0cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 -- 172.21.15.1:0/3683899900 >> v1:172.21.15.1:6789/0 conn(0x7ff8bc15e390 legacy=0x7ff8bc0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8ba7fc640 1 -- 172.21.15.1:0/3683899900 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc15e770 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.039+0000 7ff8b9ffb640 1 -- 172.21.15.1:0/3683899900 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4151192761 0 0) 0x7ff8bc0b0b70 con 0x7ff8bc15e390 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.040+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/3683899900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc08d820 tx=0x7ff8a402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95286a954c3bb442 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.040+0000 7ff8b9ffb640 1 -- 172.21.15.1:0/3683899900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4030860 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.040+0000 7ff8b9ffb640 1 -- 172.21.15.1:0/3683899900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a40309c0 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.040+0000 7ff8b9ffb640 1 -- 172.21.15.1:0/3683899900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4030d50 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 -- 172.21.15.1:0/3683899900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0d2a20 msgr2=0x7ff8bc0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/3683899900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc08d820 tx=0x7ff8a402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 -- 172.21.15.1:0/3683899900 shutdown_connections 2024-03-24T10:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/3683899900 >> v2:172.21.15.3:3300/0 conn(0x7ff8bc0b2310 0x7ff8bc0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/3683899900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0cead0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 -- 172.21.15.1:0/3683899900 >> 172.21.15.1:0/3683899900 conn(0x7ff8bc0d3ee0 msgr2=0x7ff8bc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 -- 172.21.15.1:0/3683899900 shutdown_connections 2024-03-24T10:36:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.041+0000 7ff8c12ae640 1 -- 172.21.15.1:0/3683899900 wait complete. 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 Processor -- start 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 -- start start 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 0x7ff8bc1605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc0b4690 con 0x7ff8bc0b2310 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc0b4970 con 0x7ff8bc0d2a20 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8c12ae640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff8bc0b4c50 con 0x7ff8bc15e390 2024-03-24T10:36:06.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54596/0 (socket says 172.21.15.1:54596) 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.043+0000 7ff8baffd640 1 -- 172.21.15.1:0/1828215968 learned_addr learned my addr 172.21.15.1:0/1828215968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8bb7fe640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 0x7ff8bc1605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 -- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 msgr2=0x7ff8bc1605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 0x7ff8bc1605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 -- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 msgr2=0x7ff8bc0b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 -- 172.21.15.1:0/1828215968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a40094d0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8bb7fe640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 0x7ff8bc1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8baffd640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400c7c0 tx=0x7ff8b400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400a640 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b400a7c0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc0b4f30 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400ab70 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.044+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc15d800 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.045+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8bc0b50c0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.045+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff8b400acd0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.046+0000 7ff8b37fe640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 0x7ff89c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.046+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7ff8b4089a60 con 0x7ff8bc0b2310 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.046+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 0x7ff89c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.046+0000 7ff8ba7fc640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 0x7ff89c06dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4036000 tx=0x7ff8a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.050+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff8b4054480 con 0x7ff8bc0b2310 2024-03-24T10:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.230+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff8bc04f4e0 con 0x7ff8bc0b2310 2024-03-24T10:36:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.230+0000 7ff8b37fe640 1 -- 172.21.15.1:0/1828215968 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7ff8b4018020 con 0x7ff8bc0b2310 2024-03-24T10:36:06.231 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 msgr2=0x7ff89c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 0x7ff89c06dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4036000 tx=0x7ff8a4008030 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 msgr2=0x7ff8bc15d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400c7c0 tx=0x7ff8b400cc80 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 shutdown_connections 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff89c06b810 0x7ff89c06dcd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff8bc15e390 0x7ff8bc1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8bc0d2a20 0x7ff8bc0b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 --2- 172.21.15.1:0/1828215968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8bc0b2310 0x7ff8bc15d2c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 >> 172.21.15.1:0/1828215968 conn(0x7ff8bc0d3ee0 msgr2=0x7ff8bc15e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 shutdown_connections 2024-03-24T10:36:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.232+0000 7ff8c12ae640 1 -- 172.21.15.1:0/1828215968 wait complete. 2024-03-24T10:36:06.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:36:06.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.347+0000 7f36529f4640 1 Processor -- start 2024-03-24T10:36:06.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.347+0000 7f36529f4640 1 -- start start 2024-03-24T10:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f36529f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f36529f4640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f364c0d1000 0x7f364c0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f36529f4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364c15e770 con 0x7f364c15e390 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f36529f4640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f364c0b3190 con 0x7f364c0d1000 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f36529f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f364c0d4e40 con 0x7f364c0d2bb0 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f3650f91640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f364c0d1000 0x7f364c0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f364b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f364b7fe640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:54652/0 (socket says 172.21.15.1:54652) 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f364bfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f364c15e390 0x7f364c0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:60014/0 (socket says 172.21.15.1:60014) 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.348+0000 7f364b7fe640 1 -- 172.21.15.1:0/828637600 learned_addr learned my addr 172.21.15.1:0/828637600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f3650f91640 1 -- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 msgr2=0x7f364c0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f3650f91640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f3650f91640 1 -- 172.21.15.1:0/828637600 >> v1:172.21.15.3:6789/0 conn(0x7f364c15e390 legacy=0x7f364c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f3650f91640 1 -- 172.21.15.1:0/828637600 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c0d1e40 con 0x7f364c0d1000 2024-03-24T10:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f364b7fe640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.349+0000 7f3650f91640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d1000 0x7f364c0b40d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3644009d90 tx=0x7f364402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830923150d9a3506 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f364affd640 1 -- 172.21.15.1:0/828637600 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364402cb30 con 0x7f364c0d1000 2024-03-24T10:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f364affd640 1 -- 172.21.15.1:0/828637600 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f364402cc90 con 0x7f364c0d1000 2024-03-24T10:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f364affd640 1 -- 172.21.15.1:0/828637600 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644034440 con 0x7f364c0d1000 2024-03-24T10:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 -- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d1000 msgr2=0x7f364c0b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d1000 0x7f364c0b40d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3644009d90 tx=0x7f364402bd80 comp rx=0 tx=0).stop 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 -- 172.21.15.1:0/828637600 shutdown_connections 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d1000 0x7f364c0b40d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 --2- 172.21.15.1:0/828637600 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c0d2bb0 0x7f364c0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 -- 172.21.15.1:0/828637600 >> 172.21.15.1:0/828637600 conn(0x7f364c0d3ec0 msgr2=0x7f364c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 -- 172.21.15.1:0/828637600 shutdown_connections 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.350+0000 7f36529f4640 1 -- 172.21.15.1:0/828637600 wait complete. 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.351+0000 7f36529f4640 1 Processor -- start 2024-03-24T10:36:06.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.352+0000 7f36529f4640 1 -- start start 2024-03-24T10:36:06.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d2bb0 0x7f364c1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c15e390 0x7f364c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f364c0b9520 con 0x7f364c0d1000 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f364c14b470 con 0x7f364c0d2bb0 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f36529f4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f364c0b8b80 con 0x7f364c15e390 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54604/0 (socket says 172.21.15.1:54604) 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 -- 172.21.15.1:0/471820242 learned_addr learned my addr 172.21.15.1:0/471820242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364b7fe640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d2bb0 0x7f364c1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f3650f91640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c15e390 0x7f364c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 -- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c15e390 msgr2=0x7f364c0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c15e390 0x7f364c0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 -- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d2bb0 msgr2=0x7f364c1f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d2bb0 0x7f364c1f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.353+0000 7f364bfff640 1 -- 172.21.15.1:0/471820242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c0094d0 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f364bfff640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f364000c3c0 tx=0x7f364000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36400074e0 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3640007640 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3644009a70 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640004a00 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.354+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c0b9140 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.355+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364c04f4e0 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.355+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3640004b60 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.356+0000 7f3648ff9640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 0x7f363806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.356+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f3640089850 con 0x7f364c0d1000 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.356+0000 7f364b7fe640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 0x7f363806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.356+0000 7f364b7fe640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 0x7f363806dd00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f364c161d40 tx=0x7f363c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.360+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3640054330 con 0x7f364c0d1000 2024-03-24T10:36:06.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.602+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f364c15e770 con 0x7f364c0d1000 2024-03-24T10:36:06.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.603+0000 7f3648ff9640 1 -- 172.21.15.1:0/471820242 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) ==== 80+0+0 (secure 0 0 0) 0x7f364c15e770 con 0x7f364c0d1000 2024-03-24T10:36:06.603 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:36:06.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 msgr2=0x7f363806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 0x7f363806dd00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f364c161d40 tx=0x7f363c009240 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 msgr2=0x7f364c0b9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f364000c3c0 tx=0x7f364000c880 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 shutdown_connections 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f363806b840 0x7f363806dd00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f364c15e390 0x7f364c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c0d2bb0 0x7f364c1f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 --2- 172.21.15.1:0/471820242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f364c0d1000 0x7f364c0b9940 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 >> 172.21.15.1:0/471820242 conn(0x7f364c0d3ec0 msgr2=0x7f364c0b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 shutdown_connections 2024-03-24T10:36:06.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.605+0000 7f36529f4640 1 -- 172.21.15.1:0/471820242 wait complete. 2024-03-24T10:36:06.616 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:36:06.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:06.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:06.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[57726]: fuse finished with error 0 and tester_r 0 2024-03-24T10:36:06.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:06.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:36:06.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:06.731 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:36:06.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:36:06.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.890+0000 7f7d53e34640 1 Processor -- start 2024-03-24T10:36:06.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.890+0000 7f7d53e34640 1 -- start start 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d53e34640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d53e34640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 0x7f7d4c0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d53e34640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d4c15e770 con 0x7f7d4c15e390 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d53e34640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7d4c0b3190 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d53e34640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7d4c0d4e40 con 0x7f7d4c0d1000 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d513cf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d51bd0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d4c15e390 0x7f7d4c0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40914/0 (socket says 172.21.15.1:40914) 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d513cf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54614/0 (socket says 172.21.15.1:54614) 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.891+0000 7f7d51bd0640 1 -- 172.21.15.1:0/4075057957 learned_addr learned my addr 172.21.15.1:0/4075057957 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d523d1640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 0x7f7d4c0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d513cf640 1 -- 172.21.15.1:0/4075057957 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 msgr2=0x7f7d4c0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:06.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d513cf640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 0x7f7d4c0b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d513cf640 1 -- 172.21.15.1:0/4075057957 >> v1:172.21.15.1:6789/0 conn(0x7f7d4c15e390 legacy=0x7f7d4c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d50bce640 1 -- 172.21.15.1:0/4075057957 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095682555 0 0) 0x7f7d44001110 con 0x7f7d4c15e390 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d513cf640 1 -- 172.21.15.1:0/4075057957 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4c0d1e40 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d523d1640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 0x7f7d4c0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.892+0000 7f7d513cf640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0097a0 tx=0x7f7d3c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f290918051bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d50bce640 1 -- 172.21.15.1:0/4075057957 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c02f050 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d50bce640 1 -- 172.21.15.1:0/4075057957 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d3c031700 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d50bce640 1 -- 172.21.15.1:0/4075057957 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c031a90 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 -- 172.21.15.1:0/4075057957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c0d2bb0 msgr2=0x7f7d4c0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0097a0 tx=0x7f7d3c02b7a0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 -- 172.21.15.1:0/4075057957 shutdown_connections 2024-03-24T10:36:06.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d1000 0x7f7d4c0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 --2- 172.21.15.1:0/4075057957 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c0d2bb0 0x7f7d4c0b3b90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 -- 172.21.15.1:0/4075057957 >> 172.21.15.1:0/4075057957 conn(0x7f7d4c0d3ec0 msgr2=0x7f7d4c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 -- 172.21.15.1:0/4075057957 shutdown_connections 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.893+0000 7f7d53e34640 1 -- 172.21.15.1:0/4075057957 wait complete. 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.894+0000 7f7d53e34640 1 Processor -- start 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 -- start start 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d4c0d1000 0x7f7d4c1512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d2bb0 0x7f7d4c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c0caf90 con 0x7f7d4c15e390 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c1517e0 con 0x7f7d4c0d1000 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d53e34640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7d4c0cb760 con 0x7f7d4c0d2bb0 2024-03-24T10:36:06.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d523d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d523d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54616/0 (socket says 172.21.15.1:54616) 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.895+0000 7f7d523d1640 1 -- 172.21.15.1:0/3307119608 learned_addr learned my addr 172.21.15.1:0/3307119608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d513cf640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d2bb0 0x7f7d4c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d51bd0640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d4c0d1000 0x7f7d4c1512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 -- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d2bb0 msgr2=0x7f7d4c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d2bb0 0x7f7d4c06cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 -- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d4c0d1000 msgr2=0x7f7d4c1512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d4c0d1000 0x7f7d4c1512a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 -- 172.21.15.1:0/3307119608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c009400 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d523d1640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800cea0 tx=0x7f7d4800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48003b90 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d48003d10 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48004260 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4c0cb1a0 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.896+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4c0cde20 con 0x7f7d4c15e390 2024-03-24T10:36:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.897+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4c04f4e0 con 0x7f7d4c15e390 2024-03-24T10:36:06.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.898+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7d480043c0 con 0x7f7d4c15e390 2024-03-24T10:36:06.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.898+0000 7f7d427fc640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 0x7f7d3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:06.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.898+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f7d4808aa40 con 0x7f7d4c15e390 2024-03-24T10:36:06.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.898+0000 7f7d51bd0640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 0x7f7d3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:06.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.899+0000 7f7d51bd0640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 0x7f7d3006dd00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7d44000f60 tx=0x7f7d44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:06.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:06.902+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d4808ce60 con 0x7f7d4c15e390 2024-03-24T10:36:07.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.120+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7d4c0d2280 con 0x7f7d4c15e390 2024-03-24T10:36:07.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.121+0000 7f7d427fc640 1 -- 172.21.15.1:0/3307119608 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f7d480554f0 con 0x7f7d4c15e390 2024-03-24T10:36:07.121 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 msgr2=0x7f7d3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 0x7f7d3006dd00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7d44000f60 tx=0x7f7d44009040 comp rx=0 tx=0).stop 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 msgr2=0x7f7d4c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800cea0 tx=0x7f7d4800cf90 comp rx=0 tx=0).stop 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 shutdown_connections 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7d3006b840 0x7f7d3006dd00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d4c15e390 0x7f7d4c06ba70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7d4c0d2bb0 0x7f7d4c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 --2- 172.21.15.1:0/3307119608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d4c0d1000 0x7f7d4c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.122+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 >> 172.21.15.1:0/3307119608 conn(0x7f7d4c0d3ec0 msgr2=0x7f7d4c0d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.123+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 shutdown_connections 2024-03-24T10:36:07.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07.123+0000 7f7d53e34640 1 -- 172.21.15.1:0/3307119608 wait complete. 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:36:07.134 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:07.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.135 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:36:07.149 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:36:07.149 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:36:07.150 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42573sec preferred_lft 42573sec 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:36:07.151 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.152 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:07.152 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:07.152 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:36:07.152 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:07.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.152 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:36:07.215 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:36:07.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.217 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:36:07.272 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:36:07.272 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:36:07.272 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:36:07.272 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:36:07.272 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:36:07.272 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:36:07.272 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:36:07.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:36:07.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:36:07.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:36:07.273 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:36:07.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:36:07.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:07.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.407 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:36:07.421 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:36:07.421 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:36:07.421 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:07.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:36:07.487 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:36:07.487 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:07.488 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:07.543 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:07.543 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:07.543 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:07.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:07.665 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:07.721 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:07.722 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:07.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.724 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:07.775 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:36:07.776 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:36:07.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:07.848 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:07.903 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:07.903 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:07.904 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:07.905 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:07.906 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:07.906 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:07.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:07.906 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:07.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:36:07.905+0000 7f8976438480 -1 init, newargv = 0x55a47e404830 newargc=15 2024-03-24T10:36:07.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[58691]: starting ceph client 2024-03-24T10:36:07.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[58691]: starting fuse 2024-03-24T10:36:07.921 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:36:07.922 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:36:07.922 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:07.923 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:36:07.923 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:36:07.923 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:36:07.923 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:36:07.925 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:07.926 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:36:07.926 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:36:08.042 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.58691.asok 2024-03-24T10:36:08.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:08.052 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58691.asok 2024-03-24T10:36:08.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:08.053 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58691.asok status 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "58691", 2024-03-24T10:36:08.191 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6243, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6243 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3941588077 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:08.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3941588077 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6243 192.168.0.1:0/3941588077", 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/3941588077", 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 25, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 50, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:36:08.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:08.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:08.200 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:36:08.200 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:08.215 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:36:08.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:08.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:08.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:08.281 DEBUG:teuthology.orchestra.run.smithi001:> stdin-killer -- python3 -c ' 2024-03-24T10:36:08.281 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:36:08.281 DEBUG:teuthology.orchestra.run.smithi001:> import errno 2024-03-24T10:36:08.281 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:08.281 DEBUG:teuthology.orchestra.run.smithi001:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> try: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> if False: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> ret = os.read(fd, 1024) 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> else: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> os.write(fd, b'"'"'content'"'"') 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> except OSError as e: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> if e.errno != errno.EPERM: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> raise 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> else: 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("client does not check permission of data pool") 2024-03-24T10:36:08.282 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:36:08.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:08.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-24T10:36:08.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.546+0000 7f63396c0640 1 Processor -- start 2024-03-24T10:36:08.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.546+0000 7f63396c0640 1 -- start start 2024-03-24T10:36:08.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63396c0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63341615f0 0x7f63340d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63396c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63340c7a40 con 0x7f63340b4e40 2024-03-24T10:36:08.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63396c0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63340b55b0 con 0x7f6334162610 2024-03-24T10:36:08.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63396c0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f63340cfd50 con 0x7f63341615f0 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6332ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63341615f0 0x7f63340d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63327fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f63340b4e40 0x7f6334161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40920/0 (socket says 172.21.15.1:40920) 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6332ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63341615f0 0x7f63340d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54632/0 (socket says 172.21.15.1:54632) 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f63327fc640 1 -- 172.21.15.1:0/3649956610 learned_addr learned my addr 172.21.15.1:0/3649956610 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6332ffd640 1 -- 172.21.15.1:0/3649956610 >> v1:172.21.15.3:6789/0 conn(0x7f6334162610 legacy=0x7f63340d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6332ffd640 1 -- 172.21.15.1:0/3649956610 >> v1:172.21.15.1:6789/0 conn(0x7f63340b4e40 legacy=0x7f6334161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6331ffb640 1 -- 172.21.15.1:0/3649956610 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452417590 0 0) 0x7f63340c7a40 con 0x7f63340b4e40 2024-03-24T10:36:08.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.548+0000 7f6332ffd640 1 -- 172.21.15.1:0/3649956610 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63340b52d0 con 0x7f63341615f0 2024-03-24T10:36:08.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.549+0000 7f6332ffd640 1 --2- 172.21.15.1:0/3649956610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f63340d4a10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7c3719621d5b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.549+0000 7f6331ffb640 1 -- 172.21.15.1:0/3649956610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c02cd00 con 0x7f63341615f0 2024-03-24T10:36:08.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.549+0000 7f6331ffb640 1 -- 172.21.15.1:0/3649956610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632c02ce60 con 0x7f63341615f0 2024-03-24T10:36:08.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.549+0000 7f6331ffb640 1 -- 172.21.15.1:0/3649956610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c034660 con 0x7f63341615f0 2024-03-24T10:36:08.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 -- 172.21.15.1:0/3649956610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 msgr2=0x7f63340d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 --2- 172.21.15.1:0/3649956610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f63340d4a10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02bf50 comp rx=0 tx=0).stop 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 -- 172.21.15.1:0/3649956610 shutdown_connections 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 --2- 172.21.15.1:0/3649956610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f63340d4a10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 -- 172.21.15.1:0/3649956610 >> 172.21.15.1:0/3649956610 conn(0x7f63340d3ef0 msgr2=0x7f63340d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 -- 172.21.15.1:0/3649956610 shutdown_connections 2024-03-24T10:36:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.550+0000 7f63396c0640 1 -- 172.21.15.1:0/3649956610 wait complete. 2024-03-24T10:36:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.551+0000 7f63396c0640 1 Processor -- start 2024-03-24T10:36:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.551+0000 7f63396c0640 1 -- start start 2024-03-24T10:36:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6334162610 0x7f633415b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63341f3ed0 0x7f63340cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f633415bad0 con 0x7f63341615f0 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f63341f46b0 con 0x7f63341f3ed0 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63396c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f63341f4990 con 0x7f6334162610 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54636/0 (socket says 172.21.15.1:54636) 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 -- 172.21.15.1:0/2115878477 learned_addr learned my addr 172.21.15.1:0/2115878477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63327fc640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6334162610 0x7f633415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f63337fe640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63341f3ed0 0x7f63340cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 -- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6334162610 msgr2=0x7f633415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6334162610 0x7f633415b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 -- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63341f3ed0 msgr2=0x7f63340cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63341f3ed0 0x7f63340cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 -- 172.21.15.1:0/2115878477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632c009a70 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.552+0000 7f6332ffd640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f632c037040 tx=0x7f632c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.553+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c003b90 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.553+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632c003d10 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.553+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63341f4c70 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.553+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c004270 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.553+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63341f4350 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.554+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63341f4e80 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.555+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f632c03b040 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.555+0000 7f632b7fe640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 0x7f631406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.555+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f632c0ac350 con 0x7f63341615f0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.555+0000 7f63327fc640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 0x7f631406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.555+0000 7f63327fc640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 0x7f631406dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f631c007f70 tx=0x7f631c007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.559+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f632c076eb0 con 0x7f63341615f0 2024-03-24T10:36:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.812+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) -- 0x7f633404f470 con 0x7f63341615f0 2024-03-24T10:36:08.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.815+0000 7f632b7fe640 1 -- 172.21.15.1:0/2115878477 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v19) ==== 183+0+140 (secure 0 0 0) 0x7f632c07a570 con 0x7f63341615f0 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-24T10:36:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:36:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.816+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 msgr2=0x7f631406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.816+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 0x7f631406dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f631c007f70 tx=0x7f631c007f00 comp rx=0 tx=0).stop 2024-03-24T10:36:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.816+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 msgr2=0x7f633415d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.816+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f632c037040 tx=0x7f632c004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 shutdown_connections 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f631406b810 0x7f631406dcd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63341f3ed0 0x7f63340cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6334162610 0x7f633415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 --2- 172.21.15.1:0/2115878477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63341615f0 0x7f633415d690 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 >> 172.21.15.1:0/2115878477 conn(0x7f63340d3ef0 msgr2=0x7f633406c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 shutdown_connections 2024-03-24T10:36:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.817+0000 7f63396c0640 1 -- 172.21.15.1:0/2115878477 wait complete. 2024-03-24T10:36:08.832 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:08.832 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:08.847 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:08.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.988+0000 7f7f298c0640 1 Processor -- start 2024-03-24T10:36:08.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.988+0000 7f7f298c0640 1 -- start start 2024-03-24T10:36:08.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f298c0640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7f2415e390 0x7f7f240d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f298c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f2415e770 con 0x7f7f240d1000 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f298c0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f240d4e40 con 0x7f7f240b4ba0 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f298c0640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f7f240b3190 con 0x7f7f2415e390 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f1bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7f240d1000 0x7f7f240cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40936/0 (socket says 172.21.15.1:40936) 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f1bfff640 1 -- 172.21.15.1:0/3830516230 learned_addr learned my addr 172.21.15.1:0/3830516230 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/3830516230 >> v2:172.21.15.3:3300/0 conn(0x7f7f2415e390 0x7f7f240d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145306482 0 0) 0x7f7f0c001110 con 0x7f7f240d1000 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f10003520 con 0x7f7f240d1000 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f22ffd640 1 -- 172.21.15.1:0/3830516230 >> v1:172.21.15.3:6789/0 conn(0x7f7f240b4ba0 legacy=0x7f7f240cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f22ffd640 1 -- 172.21.15.1:0/3830516230 >> v1:172.21.15.1:6789/0 conn(0x7f7f240d1000 legacy=0x7f7f240cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.989+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4212661325 0 0) 0x7f7f10003520 con 0x7f7f240d1000 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.990+0000 7f7f22ffd640 1 -- 172.21.15.1:0/3830516230 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f240d1e40 con 0x7f7f2415e390 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.990+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/3830516230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f2415e390 0x7f7f240d24a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097a0 tx=0x7f7f1402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2d1f3d8607baed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.990+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c001800 con 0x7f7f2415e390 2024-03-24T10:36:08.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1402e050 con 0x7f7f2415e390 2024-03-24T10:36:08.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f227fc640 1 -- 172.21.15.1:0/3830516230 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f14030730 con 0x7f7f2415e390 2024-03-24T10:36:08.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 -- 172.21.15.1:0/3830516230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f2415e390 msgr2=0x7f7f240d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 --2- 172.21.15.1:0/3830516230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f2415e390 0x7f7f240d24a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097a0 tx=0x7f7f1402b710 comp rx=0 tx=0).stop 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 -- 172.21.15.1:0/3830516230 shutdown_connections 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 --2- 172.21.15.1:0/3830516230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f2415e390 0x7f7f240d24a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 -- 172.21.15.1:0/3830516230 >> 172.21.15.1:0/3830516230 conn(0x7f7f240d3ec0 msgr2=0x7f7f240d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 -- 172.21.15.1:0/3830516230 shutdown_connections 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.991+0000 7f7f298c0640 1 -- 172.21.15.1:0/3830516230 wait complete. 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.992+0000 7f7f298c0640 1 Processor -- start 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.992+0000 7f7f298c0640 1 -- start start 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 0x7f7f2414fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f240d1000 0x7f7f2415b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2415c100 con 0x7f7f2415e390 2024-03-24T10:36:08.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240cc520 con 0x7f7f240d1000 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f298c0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f7f240cc800 con 0x7f7f240b4ba0 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f237fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54650/0 (socket says 172.21.15.1:54650) 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f237fe640 1 -- 172.21.15.1:0/1735970319 learned_addr learned my addr 172.21.15.1:0/1735970319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 0x7f7f2414fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f1bfff640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f240d1000 0x7f7f2415b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.993+0000 7f7f237fe640 1 -- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 msgr2=0x7f7f2414fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f237fe640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 0x7f7f2414fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f237fe640 1 -- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f240d1000 msgr2=0x7f7f2415b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f237fe640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f240d1000 0x7f7f2415b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f237fe640 1 -- 172.21.15.1:0/1735970319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f14009400 con 0x7f7f2415e390 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 0x7f7f2414fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:08.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f237fe640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00c6b0 tx=0x7f7f1c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c003b70 con 0x7f7f2415e390 2024-03-24T10:36:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f240ccae0 con 0x7f7f2415e390 2024-03-24T10:36:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c003cf0 con 0x7f7f2415e390 2024-03-24T10:36:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c003ea0 con 0x7f7f2415e390 2024-03-24T10:36:08.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.994+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f2415bd60 con 0x7f7f2415e390 2024-03-24T10:36:08.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.995+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f2404f4e0 con 0x7f7f2415e390 2024-03-24T10:36:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.995+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f1c005310 con 0x7f7f2415e390 2024-03-24T10:36:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.996+0000 7f7f20ff9640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 0x7f7f0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.996+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f7f1c089fd0 con 0x7f7f2415e390 2024-03-24T10:36:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.996+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 0x7f7f0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:08.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:08.996+0000 7f7f22ffd640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 0x7f7f0406dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7f14034000 tx=0x7f7f14008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.000+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f1c054970 con 0x7f7f2415e390 2024-03-24T10:36:09.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.182+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f240d2280 con 0x7f7f2415e390 2024-03-24T10:36:09.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.184+0000 7f7f20ff9640 1 -- 172.21.15.1:0/1735970319 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f7f1c058030 con 0x7f7f2415e390 2024-03-24T10:36:09.185 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:09.185 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 msgr2=0x7f7f0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 0x7f7f0406dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7f14034000 tx=0x7f7f14008030 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 msgr2=0x7f7f240b8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00c6b0 tx=0x7f7f1c00cb70 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 shutdown_connections 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f7f0406b810 0x7f7f0406dcd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f2415e390 0x7f7f240b8cf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f240d1000 0x7f7f2415b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 --2- 172.21.15.1:0/1735970319 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f7f240b4ba0 0x7f7f2414fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 >> 172.21.15.1:0/1735970319 conn(0x7f7f240d3ec0 msgr2=0x7f7f240b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 shutdown_connections 2024-03-24T10:36:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.186+0000 7f7f298c0640 1 -- 172.21.15.1:0/1735970319 wait complete. 2024-03-24T10:36:09.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:09.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:09.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.303+0000 7faf5630c640 1 Processor -- start 2024-03-24T10:36:09.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.303+0000 7faf5630c640 1 -- start start 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf5630c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf5015e390 0x7faf500d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf5630c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faf500cfae0 0x7faf500d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf5630c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf500b0b70 con 0x7faf500b2ed0 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf5630c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7faf500d1d60 con 0x7faf5015e390 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf5630c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7faf500b0de0 con 0x7faf500cfae0 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf5015e390 0x7faf500d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faf500b2ed0 0x7faf500b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40938/0 (socket says 172.21.15.1:40938) 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4f7fe640 1 -- 172.21.15.1:0/1434379882 learned_addr learned my addr 172.21.15.1:0/1434379882 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 --2- 172.21.15.1:0/1434379882 >> v2:172.21.15.1:3300/0 conn(0x7faf5015e390 0x7faf500d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54656/0 (socket says 172.21.15.1:54656) 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf548a9640 1 --2- 172.21.15.1:0/1434379882 >> v2:172.21.15.3:3300/0 conn(0x7faf500cfae0 0x7faf500d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 -- 172.21.15.1:0/1434379882 >> v2:172.21.15.3:3300/0 conn(0x7faf500cfae0 msgr2=0x7faf500d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 --2- 172.21.15.1:0/1434379882 >> v2:172.21.15.3:3300/0 conn(0x7faf500cfae0 0x7faf500d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 -- 172.21.15.1:0/1434379882 >> v1:172.21.15.1:6789/0 conn(0x7faf500b2ed0 legacy=0x7faf500b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4ffff640 1 -- 172.21.15.1:0/1434379882 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5015e770 con 0x7faf5015e390 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.304+0000 7faf4effd640 1 -- 172.21.15.1:0/1434379882 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112114354 0 0) 0x7faf500b0b70 con 0x7faf500b2ed0 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.305+0000 7faf4ffff640 1 --2- 172.21.15.1:0/1434379882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5015e390 0x7faf500d2310 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7faf5008d820 tx=0x7faf4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729e9be24b9fb4ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.305+0000 7faf4effd640 1 -- 172.21.15.1:0/1434379882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40031830 con 0x7faf5015e390 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.305+0000 7faf4effd640 1 -- 172.21.15.1:0/1434379882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf40031990 con 0x7faf5015e390 2024-03-24T10:36:09.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.305+0000 7faf4effd640 1 -- 172.21.15.1:0/1434379882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40031d20 con 0x7faf5015e390 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 -- 172.21.15.1:0/1434379882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5015e390 msgr2=0x7faf500d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 --2- 172.21.15.1:0/1434379882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5015e390 0x7faf500d2310 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7faf5008d820 tx=0x7faf4002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 -- 172.21.15.1:0/1434379882 shutdown_connections 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 --2- 172.21.15.1:0/1434379882 >> v2:172.21.15.3:3300/0 conn(0x7faf500cfae0 0x7faf500d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 --2- 172.21.15.1:0/1434379882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5015e390 0x7faf500d2310 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 -- 172.21.15.1:0/1434379882 >> 172.21.15.1:0/1434379882 conn(0x7faf500d3ee0 msgr2=0x7faf500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 -- 172.21.15.1:0/1434379882 shutdown_connections 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.306+0000 7faf5630c640 1 -- 172.21.15.1:0/1434379882 wait complete. 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.307+0000 7faf5630c640 1 Processor -- start 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.307+0000 7faf5630c640 1 -- start start 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 0x7faf50157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 0x7faf500b9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faf50157e90 con 0x7faf500b2ed0 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7faf500b6200 con 0x7faf500cfae0 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf5630c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faf500b5d70 con 0x7faf5015e390 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54664/0 (socket says 172.21.15.1:54664) 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 -- 172.21.15.1:0/2591495619 learned_addr learned my addr 172.21.15.1:0/2591495619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf548a9640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 0x7faf500b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4f7fe640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 0x7faf50157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 -- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 msgr2=0x7faf500b9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 0x7faf500b9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 -- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 msgr2=0x7faf50157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 0x7faf50157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 -- 172.21.15.1:0/2591495619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf40009400 con 0x7faf500b2ed0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4f7fe640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 0x7faf50157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf548a9640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 0x7faf500b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.308+0000 7faf4ffff640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7faf4002bee0 tx=0x7faf400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.309+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4003a8f0 con 0x7faf500b2ed0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.309+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4003aa70 con 0x7faf500b2ed0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.309+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4003ae20 con 0x7faf500b2ed0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.309+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf500b5160 con 0x7faf500b2ed0 2024-03-24T10:36:09.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.309+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf500b5720 con 0x7faf500b2ed0 2024-03-24T10:36:09.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.310+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5004f4e0 con 0x7faf500b2ed0 2024-03-24T10:36:09.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.310+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faf400443f0 con 0x7faf500b2ed0 2024-03-24T10:36:09.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.311+0000 7faf4cff9640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 0x7faf3c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.311+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7faf400ac220 con 0x7faf500b2ed0 2024-03-24T10:36:09.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.311+0000 7faf4f7fe640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 0x7faf3c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.311+0000 7faf4f7fe640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 0x7faf3c06dd00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faf5015ac00 tx=0x7faf44007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.315+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faf40076c40 con 0x7faf500b2ed0 2024-03-24T10:36:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.495+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf500d3210 con 0x7faf500b2ed0 2024-03-24T10:36:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.495+0000 7faf4cff9640 1 -- 172.21.15.1:0/2591495619 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7faf400350a0 con 0x7faf500b2ed0 2024-03-24T10:36:09.496 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 msgr2=0x7faf3c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 0x7faf3c06dd00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faf5015ac00 tx=0x7faf44007c10 comp rx=0 tx=0).stop 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 msgr2=0x7faf501f3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7faf4002bee0 tx=0x7faf400046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 shutdown_connections 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf3c06b840 0x7faf3c06dd00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf5015e390 0x7faf500b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf500cfae0 0x7faf50157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 --2- 172.21.15.1:0/2591495619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf500b2ed0 0x7faf501f3780 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 >> 172.21.15.1:0/2591495619 conn(0x7faf500d3ee0 msgr2=0x7faf500b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.497+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 shutdown_connections 2024-03-24T10:36:09.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.498+0000 7faf5630c640 1 -- 172.21.15.1:0/2591495619 wait complete. 2024-03-24T10:36:09.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.613+0000 7ff99b59e640 1 Processor -- start 2024-03-24T10:36:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.613+0000 7ff99b59e640 1 -- start start 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.613+0000 7ff99b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff99c0d2270 0x7ff99c0d2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 0x7ff99c1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99b59e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99c15f0b0 con 0x7ff99c0d45b0 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99b59e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff99c0d2b90 con 0x7ff99c0d2270 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99b59e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff99c0d4b20 con 0x7ff99c0d0ed0 2024-03-24T10:36:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff99c0d2270 0x7ff99c0d2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff999d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff99c0d45b0 0x7ff99c161be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40942/0 (socket says 172.21.15.1:40942) 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff999d9b640 1 -- 172.21.15.1:0/1218528970 learned_addr learned my addr 172.21.15.1:0/1218528970 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99ad9d640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 0x7ff99c1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99a59c640 1 -- 172.21.15.1:0/1218528970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 msgr2=0x7ff99c1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99a59c640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 0x7ff99c1622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99a59c640 1 -- 172.21.15.1:0/1218528970 >> v1:172.21.15.1:6789/0 conn(0x7ff99c0d45b0 legacy=0x7ff99c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99959a640 1 -- 172.21.15.1:0/1218528970 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606745544 0 0) 0x7ff99c15f0b0 con 0x7ff99c0d45b0 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99a59c640 1 -- 172.21.15.1:0/1218528970 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c15e780 con 0x7ff99c0d2270 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.614+0000 7ff99ad9d640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 0x7ff99c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99a59c640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c0d2650 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0097a0 tx=0x7ff98c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82db2d0ffba63b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99959a640 1 -- 172.21.15.1:0/1218528970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c031830 con 0x7ff99c0d2270 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99959a640 1 -- 172.21.15.1:0/1218528970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff98c031990 con 0x7ff99c0d2270 2024-03-24T10:36:09.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99959a640 1 -- 172.21.15.1:0/1218528970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c031d20 con 0x7ff99c0d2270 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99b59e640 1 -- 172.21.15.1:0/1218528970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 msgr2=0x7ff99c0d2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.615+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c0d2650 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0097a0 tx=0x7ff98c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 -- 172.21.15.1:0/1218528970 shutdown_connections 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d0ed0 0x7ff99c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1218528970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c0d2650 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 -- 172.21.15.1:0/1218528970 >> 172.21.15.1:0/1218528970 conn(0x7ff99c09d000 msgr2=0x7ff99c0d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 -- 172.21.15.1:0/1218528970 shutdown_connections 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.616+0000 7ff99b59e640 1 -- 172.21.15.1:0/1218528970 wait complete. 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.617+0000 7ff99b59e640 1 Processor -- start 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.617+0000 7ff99b59e640 1 -- start start 2024-03-24T10:36:09.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0d0ed0 0x7ff99c1f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 0x7ff99c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c1f8740 con 0x7ff99c0d2270 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c0b4f30 con 0x7ff99c0d0ed0 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ff99c0b51d0 con 0x7ff99c0d45b0 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54674/0 (socket says 172.21.15.1:54674) 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 -- 172.21.15.1:0/1008504134 learned_addr learned my addr 172.21.15.1:0/1008504134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99ad9d640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 0x7ff99c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99a59c640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0d0ed0 0x7ff99c1f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 -- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 msgr2=0x7ff99c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 0x7ff99c15a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 -- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0d0ed0 msgr2=0x7ff99c1f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0d0ed0 0x7ff99c1f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 -- 172.21.15.1:0/1008504134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c009400 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99ad9d640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 0x7ff99c15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff999d9b640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c430 tx=0x7ff99000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9900074e0 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff990007660 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990004a20 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.618+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c0b54b0 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.619+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c0b5a70 con 0x7ff99c0d2270 2024-03-24T10:36:09.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.620+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c15f520 con 0x7ff99c0d2270 2024-03-24T10:36:09.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.620+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff990004b80 con 0x7ff99c0d2270 2024-03-24T10:36:09.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.621+0000 7ff98affd640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 0x7ff97c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.621+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7ff990089f60 con 0x7ff99c0d2270 2024-03-24T10:36:09.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.621+0000 7ff99a59c640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 0x7ff97c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.621+0000 7ff99a59c640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 0x7ff97c06dcd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036000 tx=0x7ff98c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.625+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff99c15f520 con 0x7ff99c0d2270 2024-03-24T10:36:09.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.803+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff99c15f520 con 0x7ff99c0d2270 2024-03-24T10:36:09.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.804+0000 7ff98affd640 1 -- 172.21.15.1:0/1008504134 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7ff99c15f520 con 0x7ff99c0d2270 2024-03-24T10:36:09.804 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:09.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 msgr2=0x7ff97c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 0x7ff97c06dcd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036000 tx=0x7ff98c03c040 comp rx=0 tx=0).stop 2024-03-24T10:36:09.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 msgr2=0x7ff99c14cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c430 tx=0x7ff99000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 shutdown_connections 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ff97c06b810 0x7ff97c06dcd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ff99c0d45b0 0x7ff99c15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff99c0d2270 0x7ff99c14cee0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 --2- 172.21.15.1:0/1008504134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0d0ed0 0x7ff99c1f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 >> 172.21.15.1:0/1008504134 conn(0x7ff99c09d000 msgr2=0x7ff99c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 shutdown_connections 2024-03-24T10:36:09.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.806+0000 7ff99b59e640 1 -- 172.21.15.1:0/1008504134 wait complete. 2024-03-24T10:36:09.817 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:09.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.925+0000 7fb133084640 1 Processor -- start 2024-03-24T10:36:09.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.926+0000 7fb133084640 1 -- start start 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.926+0000 7fb133084640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb12c0d2a20 0x7fb12c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.926+0000 7fb133084640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12c0b0b70 con 0x7fb12c15e390 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.926+0000 7fb133084640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12c0d1d60 con 0x7fb12c0b2310 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.926+0000 7fb133084640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb12c0b0de0 con 0x7fb12c0d2a20 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb12c0d2a20 0x7fb12c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb130e20640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb12c15e390 0x7fb12c0d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40944/0 (socket says 172.21.15.1:40944) 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb130e20640 1 -- 172.21.15.1:0/3178369630 learned_addr learned my addr 172.21.15.1:0/3178369630 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12b7fe640 1 -- 172.21.15.1:0/3178369630 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773537441 0 0) 0x7fb12c0b0b70 con 0x7fb12c15e390 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12b7fe640 1 -- 172.21.15.1:0/3178369630 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb11c003520 con 0x7fb12c15e390 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12bfff640 1 -- 172.21.15.1:0/3178369630 >> v1:172.21.15.3:6789/0 conn(0x7fb12c0b2310 legacy=0x7fb12c0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12bfff640 1 -- 172.21.15.1:0/3178369630 >> v1:172.21.15.1:6789/0 conn(0x7fb12c15e390 legacy=0x7fb12c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.927+0000 7fb12bfff640 1 -- 172.21.15.1:0/3178369630 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c15e770 con 0x7fb12c0d2a20 2024-03-24T10:36:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.928+0000 7fb12bfff640 1 --2- 172.21.15.1:0/3178369630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c0d2a20 0x7fb12c0cead0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb1180097a0 tx=0x7fb11802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a027cb46288845b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.928+0000 7fb12b7fe640 1 -- 172.21.15.1:0/3178369630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118030610 con 0x7fb12c0d2a20 2024-03-24T10:36:09.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.928+0000 7fb12b7fe640 1 -- 172.21.15.1:0/3178369630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb118030770 con 0x7fb12c0d2a20 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.928+0000 7fb12b7fe640 1 -- 172.21.15.1:0/3178369630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118030ae0 con 0x7fb12c0d2a20 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 -- 172.21.15.1:0/3178369630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c0d2a20 msgr2=0x7fb12c0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 --2- 172.21.15.1:0/3178369630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c0d2a20 0x7fb12c0cead0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb1180097a0 tx=0x7fb11802b960 comp rx=0 tx=0).stop 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 -- 172.21.15.1:0/3178369630 shutdown_connections 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 --2- 172.21.15.1:0/3178369630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c0d2a20 0x7fb12c0cead0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 -- 172.21.15.1:0/3178369630 >> 172.21.15.1:0/3178369630 conn(0x7fb12c0d3ee0 msgr2=0x7fb12c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 -- 172.21.15.1:0/3178369630 shutdown_connections 2024-03-24T10:36:09.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.929+0000 7fb133084640 1 -- 172.21.15.1:0/3178369630 wait complete. 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.930+0000 7fb133084640 1 Processor -- start 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.930+0000 7fb133084640 1 -- start start 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 0x7fb12c14dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb12c0d2a20 0x7fb12c14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c0c9800 con 0x7fb12c15e390 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c14e1e0 con 0x7fb12c0d2a20 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb133084640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb12c0c8b80 con 0x7fb12c0b2310 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54680/0 (socket says 172.21.15.1:54680) 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 -- 172.21.15.1:0/137959902 learned_addr learned my addr 172.21.15.1:0/137959902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb130e20640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 0x7fb12c14dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb12bfff640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb12c0d2a20 0x7fb12c14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 -- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 msgr2=0x7fb12c14dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 0x7fb12c14dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 -- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb12c0d2a20 msgr2=0x7fb12c14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb12c0d2a20 0x7fb12c14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb131621640 1 -- 172.21.15.1:0/137959902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb118009400 con 0x7fb12c15e390 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.931+0000 7fb130e20640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 0x7fb12c14dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb131621640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c730 tx=0x7fb12400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124003b70 con 0x7fb12c15e390 2024-03-24T10:36:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb124003cd0 con 0x7fb12c15e390 2024-03-24T10:36:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c0c8e60 con 0x7fb12c15e390 2024-03-24T10:36:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12400aa90 con 0x7fb12c15e390 2024-03-24T10:36:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.932+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c0c9420 con 0x7fb12c15e390 2024-03-24T10:36:09.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.933+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c0d13e0 con 0x7fb12c15e390 2024-03-24T10:36:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.934+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb12400abf0 con 0x7fb12c15e390 2024-03-24T10:36:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.934+0000 7fb1297fa640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.934+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fb124089fb0 con 0x7fb12c15e390 2024-03-24T10:36:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.934+0000 7fb130e20640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:09.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.935+0000 7fb130e20640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 0x7fb11006dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1200011e0 tx=0x7fb120009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:09.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:09.938+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1240bde10 con 0x7fb12c15e390 2024-03-24T10:36:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.121+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb12c04f4e0 con 0x7fb12c15e390 2024-03-24T10:36:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.121+0000 7fb1297fa640 1 -- 172.21.15.1:0/137959902 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7fb124054940 con 0x7fb12c15e390 2024-03-24T10:36:10.121 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 msgr2=0x7fb11006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 0x7fb11006dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1200011e0 tx=0x7fb120009040 comp rx=0 tx=0).stop 2024-03-24T10:36:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 msgr2=0x7fb12c0c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c730 tx=0x7fb12400cbf0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 shutdown_connections 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb12c15e390 0x7fb12c0c9c20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb12c0d2a20 0x7fb12c14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 --2- 172.21.15.1:0/137959902 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb12c0b2310 0x7fb12c14dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 >> 172.21.15.1:0/137959902 conn(0x7fb12c0d3ee0 msgr2=0x7fb12c0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 shutdown_connections 2024-03-24T10:36:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.123+0000 7fb133084640 1 -- 172.21.15.1:0/137959902 wait complete. 2024-03-24T10:36:10.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:36:10.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.246+0000 7f2192e2a640 1 Processor -- start 2024-03-24T10:36:10.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.246+0000 7f2192e2a640 1 -- start start 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2192e2a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 0x7f218c0cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2192e2a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f218c0cff00 0x7f218c165a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2192e2a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f218c06ce40 con 0x7f218c15bda0 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2192e2a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f218c0b2db0 con 0x7f218c0cff00 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2192e2a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f218c06e140 con 0x7f218c06c910 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f218bfff640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 0x7f218c0cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f21913c7640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f218c0cff00 0x7f218c165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2190bc6640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f218c15bda0 0x7f218c15ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:60058/0 (socket says 172.21.15.1:60058) 2024-03-24T10:36:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f2190bc6640 1 -- 172.21.15.1:0/385310609 learned_addr learned my addr 172.21.15.1:0/385310609 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f21913c7640 1 -- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 msgr2=0x7f218c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f21913c7640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 0x7f218c0cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f21913c7640 1 -- 172.21.15.1:0/385310609 >> v1:172.21.15.3:6789/0 conn(0x7f218c15bda0 legacy=0x7f218c15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f21913c7640 1 -- 172.21.15.1:0/385310609 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c06d120 con 0x7f218c0cff00 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f218b7fe640 1 -- 172.21.15.1:0/385310609 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522284092 0 0) 0x7f218c06ce40 con 0x7f218c15bda0 2024-03-24T10:36:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.247+0000 7f218bfff640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 0x7f218c0cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.248+0000 7f21913c7640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c0cff00 0x7f218c165a20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2184009d90 tx=0x7f218402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8a0693dd43ca507 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f218b7fe640 1 -- 172.21.15.1:0/385310609 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218402cbc0 con 0x7f218c0cff00 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f218b7fe640 1 -- 172.21.15.1:0/385310609 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f218402cd20 con 0x7f218c0cff00 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f218b7fe640 1 -- 172.21.15.1:0/385310609 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184034660 con 0x7f218c0cff00 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 -- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c0cff00 msgr2=0x7f218c165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c0cff00 0x7f218c165a20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2184009d90 tx=0x7f218402be10 comp rx=0 tx=0).stop 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 -- 172.21.15.1:0/385310609 shutdown_connections 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c0cff00 0x7f218c165a20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 --2- 172.21.15.1:0/385310609 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c06c910 0x7f218c0cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 -- 172.21.15.1:0/385310609 >> 172.21.15.1:0/385310609 conn(0x7f218c0b17a0 msgr2=0x7f218c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 -- 172.21.15.1:0/385310609 shutdown_connections 2024-03-24T10:36:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.249+0000 7f2192e2a640 1 -- 172.21.15.1:0/385310609 wait complete. 2024-03-24T10:36:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.250+0000 7f2192e2a640 1 Processor -- start 2024-03-24T10:36:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.250+0000 7f2192e2a640 1 -- start start 2024-03-24T10:36:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c0cff00 0x7f218c148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 0x7f218c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f218c150960 con 0x7f218c06c910 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f218c150c40 con 0x7f218c15bda0 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2192e2a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f218c150f20 con 0x7f218c0cff00 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2190bc6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2190bc6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54688/0 (socket says 172.21.15.1:54688) 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.251+0000 7f2190bc6640 1 -- 172.21.15.1:0/2893516955 learned_addr learned my addr 172.21.15.1:0/2893516955 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f21913c7640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 0x7f218c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f218bfff640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c0cff00 0x7f218c148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 -- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c0cff00 msgr2=0x7f218c148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c0cff00 0x7f218c148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 -- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 msgr2=0x7f218c0ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 0x7f218c0ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 -- 172.21.15.1:0/2893516955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2178009400 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f21913c7640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 0x7f218c0ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2190bc6640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f218000c430 tx=0x7f218000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800074e0 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2184009a70 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2180007660 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180004a20 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.252+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218c1514e0 con 0x7f218c06c910 2024-03-24T10:36:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.253+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218c068680 con 0x7f218c06c910 2024-03-24T10:36:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.254+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2180004b80 con 0x7f218c06c910 2024-03-24T10:36:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.254+0000 7f21897fa640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 0x7f217006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.254+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f218008a100 con 0x7f218c06c910 2024-03-24T10:36:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.254+0000 7f218bfff640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 0x7f217006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.255+0000 7f218bfff640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 0x7f217006dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f218c0720a0 tx=0x7f2178008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.263+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2180054a00 con 0x7f218c06c910 2024-03-24T10:36:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.458+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f218c04f4e0 con 0x7f218c06c910 2024-03-24T10:36:10.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.458+0000 7f21897fa640 1 -- 172.21.15.1:0/2893516955 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) ==== 80+0+0 (secure 0 0 0) 0x7f21800580c0 con 0x7f218c06c910 2024-03-24T10:36:10.459 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 msgr2=0x7f217006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 0x7f217006dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f218c0720a0 tx=0x7f2178008040 comp rx=0 tx=0).stop 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 msgr2=0x7f218c14d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f218000c430 tx=0x7f218000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 shutdown_connections 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.460+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f217006b810 0x7f217006dcd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f218c15bda0 0x7f218c0ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f218c0cff00 0x7f218c148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 --2- 172.21.15.1:0/2893516955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f218c06c910 0x7f218c14d090 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 >> 172.21.15.1:0/2893516955 conn(0x7f218c0b17a0 msgr2=0x7f218c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:10.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 shutdown_connections 2024-03-24T10:36:10.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.461+0000 7f2192e2a640 1 -- 172.21.15.1:0/2893516955 wait complete. 2024-03-24T10:36:10.472 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:10.472 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:10.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:10.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.627+0000 7fbd8b2ed640 1 Processor -- start 2024-03-24T10:36:10.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.628+0000 7fbd8b2ed640 1 -- start start 2024-03-24T10:36:10.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.628+0000 7fbd8b2ed640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbd840d1000 0x7fbd840b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.628+0000 7fbd8b2ed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8415e770 con 0x7fbd840d2bb0 2024-03-24T10:36:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.628+0000 7fbd8b2ed640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd840d4e40 con 0x7fbd8415e390 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.628+0000 7fbd8b2ed640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fbd840b3190 con 0x7fbd840d1000 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd88888640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbd840d2bb0 0x7fbd840b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37442/0 (socket says 172.21.15.1:37442) 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd88888640 1 -- 172.21.15.1:0/3134179190 learned_addr learned my addr 172.21.15.1:0/3134179190 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd8988a640 1 --2- 172.21.15.1:0/3134179190 >> v2:172.21.15.3:3300/0 conn(0x7fbd840d1000 0x7fbd840b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141347627 0 0) 0x7fbd74001110 con 0x7fbd840d2bb0 2024-03-24T10:36:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd70003520 con 0x7fbd840d2bb0 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1396562819 0 0) 0x7fbd70003520 con 0x7fbd840d2bb0 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 >> v2:172.21.15.3:3300/0 conn(0x7fbd840d1000 msgr2=0x7fbd840b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 --2- 172.21.15.1:0/3134179190 >> v2:172.21.15.3:3300/0 conn(0x7fbd840d1000 0x7fbd840b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 >> v1:172.21.15.3:6789/0 conn(0x7fbd8415e390 legacy=0x7fbd840d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd840d1e40 con 0x7fbd840d2bb0 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fbd74001800 con 0x7fbd840d2bb0 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 >> v1:172.21.15.1:6789/0 conn(0x7fbd840d2bb0 legacy=0x7fbd840b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd70003970 0x7fbd70003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd700047c0 0x7fbd70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7000c0a0 con 0x7fbd70005180 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7000c210 con 0x7fbd70003970 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbd7000c3d0 con 0x7fbd700047c0 2024-03-24T10:36:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd8988a640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd89089640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd70003970 0x7fbd70003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.629+0000 7fbd88888640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd700047c0 0x7fbd70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 -- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd700047c0 msgr2=0x7fbd70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd700047c0 0x7fbd70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 -- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd70003970 msgr2=0x7fbd70003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd70003970 0x7fbd70003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 -- 172.21.15.1:0/3134179190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd70003520 con 0x7fbd70005180 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd89089640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd70003970 0x7fbd70003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd88888640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd700047c0 0x7fbd70004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 -- 172.21.15.1:0/3134179190 reap_dead start 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.630+0000 7fbd8988a640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd7000b980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009fc0 tx=0x7fbd700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16396e114fed1e02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c02cc60 con 0x7fbd70005180 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd7c00f040 con 0x7fbd70005180 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd7bfff640 1 -- 172.21.15.1:0/3134179190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c003990 con 0x7fbd70005180 2024-03-24T10:36:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 msgr2=0x7fbd7000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd7000b980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009fc0 tx=0x7fbd700057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/3134179190 shutdown_connections 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/3134179190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd7000b980 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/3134179190 >> 172.21.15.1:0/3134179190 conn(0x7fbd840d3ec0 msgr2=0x7fbd840d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/3134179190 shutdown_connections 2024-03-24T10:36:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.631+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/3134179190 wait complete. 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.632+0000 7fbd8b2ed640 1 Processor -- start 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.632+0000 7fbd8b2ed640 1 -- start start 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd841f46e0 0x7fbd840cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd8406cd30 0x7fbd841520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd841f4b60 con 0x7fbd70005180 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbd841f7230 con 0x7fbd841f46e0 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbd840cd6a0 con 0x7fbd8406cd30 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37238/0 (socket says 172.21.15.1:37238) 2024-03-24T10:36:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 -- 172.21.15.1:0/749197622 learned_addr learned my addr 172.21.15.1:0/749197622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd88888640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd841f46e0 0x7fbd840cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8988a640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd8406cd30 0x7fbd841520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 -- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd8406cd30 msgr2=0x7fbd841520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd8406cd30 0x7fbd841520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 -- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd841f46e0 msgr2=0x7fbd840cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd841f46e0 0x7fbd840cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 -- 172.21.15.1:0/749197622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd840cd940 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd89089640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e7d0 tx=0x7fbd8000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80017070 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd800050d0 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd841f74d0 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.634+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80005240 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.635+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd840cdc20 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.636+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd840cddb0 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.636+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbd80007020 con 0x7fbd70005180 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.636+0000 7fbd79ffb640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 0x7fbd6806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:10.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.637+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fbd800898c0 con 0x7fbd70005180 2024-03-24T10:36:10.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.637+0000 7fbd88888640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 0x7fbd6806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:10.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.637+0000 7fbd88888640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 0x7fbd6806dd00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbd74001270 tx=0x7fbd74007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:10.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.641+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbd80054390 con 0x7fbd70005180 2024-03-24T10:36:10.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.947+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd841f79f0 con 0x7fbd70005180 2024-03-24T10:36:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.949+0000 7fbd79ffb640 1 -- 172.21.15.1:0/749197622 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7fbd841f79f0 con 0x7fbd70005180 2024-03-24T10:36:10.949 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:10.949 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 msgr2=0x7fbd6806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 0x7fbd6806dd00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbd74001270 tx=0x7fbd74007040 comp rx=0 tx=0).stop 2024-03-24T10:36:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 msgr2=0x7fbd841f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e7d0 tx=0x7fbd8000ec90 comp rx=0 tx=0).stop 2024-03-24T10:36:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 shutdown_connections 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbd6806b840 0x7fbd6806dd00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbd8406cd30 0x7fbd841520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd841f46e0 0x7fbd840cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 --2- 172.21.15.1:0/749197622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005180 0x7fbd841f7cd0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 >> 172.21.15.1:0/749197622 conn(0x7fbd840d3ec0 msgr2=0x7fbd84156850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 shutdown_connections 2024-03-24T10:36:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:10.951+0000 7fbd8b2ed640 1 -- 172.21.15.1:0/749197622 wait complete. 2024-03-24T10:36:10.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:10.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:11.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.067+0000 7f0c8809d640 1 Processor -- start 2024-03-24T10:36:11.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.067+0000 7f0c8809d640 1 -- start start 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8809d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c8015e390 0x7f0c800d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8809d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c800b2ed0 0x7f0c800d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8809d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c800b0b70 con 0x7f0c800b1300 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8809d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0c800d1d60 con 0x7f0c800b2ed0 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8809d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0c800b0de0 con 0x7f0c8015e390 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c85638640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0c800b2ed0 0x7f0c800d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8663a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c800b1300 0x7f0c800b1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37450/0 (socket says 172.21.15.1:37450) 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c8663a640 1 -- 172.21.15.1:0/4063900975 learned_addr learned my addr 172.21.15.1:0/4063900975 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c85e39640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c8015e390 0x7f0c800d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c85638640 1 -- 172.21.15.1:0/4063900975 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c8015e390 msgr2=0x7f0c800d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:11.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c85638640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c8015e390 0x7f0c800d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c85638640 1 -- 172.21.15.1:0/4063900975 >> v1:172.21.15.1:6789/0 conn(0x7f0c800b1300 legacy=0x7f0c800b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.069+0000 7f0c85638640 1 -- 172.21.15.1:0/4063900975 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8015e770 con 0x7f0c800b2ed0 2024-03-24T10:36:11.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.068+0000 7f0c84e37640 1 -- 172.21.15.1:0/4063900975 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667472046 0 0) 0x7f0c800b0b70 con 0x7f0c800b1300 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.069+0000 7f0c85638640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b2ed0 0x7f0c800d2850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0097a0 tx=0x7f0c7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19751cf849ade519 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.069+0000 7f0c84e37640 1 -- 172.21.15.1:0/4063900975 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c031830 con 0x7f0c800b2ed0 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.069+0000 7f0c84e37640 1 -- 172.21.15.1:0/4063900975 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c7c031990 con 0x7f0c800b2ed0 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.069+0000 7f0c84e37640 1 -- 172.21.15.1:0/4063900975 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c031d20 con 0x7f0c800b2ed0 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 -- 172.21.15.1:0/4063900975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b2ed0 msgr2=0x7f0c800d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b2ed0 0x7f0c800d2850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0097a0 tx=0x7f0c7c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 -- 172.21.15.1:0/4063900975 shutdown_connections 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b2ed0 0x7f0c800d2850 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 --2- 172.21.15.1:0/4063900975 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c8015e390 0x7f0c800d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 -- 172.21.15.1:0/4063900975 >> 172.21.15.1:0/4063900975 conn(0x7f0c800d3ee0 msgr2=0x7f0c800d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 -- 172.21.15.1:0/4063900975 shutdown_connections 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.070+0000 7f0c8809d640 1 -- 172.21.15.1:0/4063900975 wait complete. 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.071+0000 7f0c8809d640 1 Processor -- start 2024-03-24T10:36:11.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.071+0000 7f0c8809d640 1 -- start start 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c800b2ed0 0x7f0c800b9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 0x7f0c801f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80150c80 con 0x7f0c800b1300 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0c800b9a40 con 0x7f0c8015e390 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8809d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0c8014fc70 con 0x7f0c800b2ed0 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37250/0 (socket says 172.21.15.1:37250) 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 -- 172.21.15.1:0/535807572 learned_addr learned my addr 172.21.15.1:0/535807572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85638640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c800b2ed0 0x7f0c800b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8663a640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 0x7f0c801f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 -- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c800b2ed0 msgr2=0x7f0c800b9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c800b2ed0 0x7f0c800b9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 -- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 msgr2=0x7f0c801f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 0x7f0c801f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 -- 172.21.15.1:0/535807572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c009400 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c8663a640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 0x7f0c801f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.072+0000 7f0c85e39640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000d5f0 tx=0x7f0c7000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.073+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c700125f0 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.073+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c70012750 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.073+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70012b00 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.073+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c801f4550 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.073+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8014ff40 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.074+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8004f4e0 con 0x7f0c800b1300 2024-03-24T10:36:11.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.074+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c70012c60 con 0x7f0c800b1300 2024-03-24T10:36:11.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.074+0000 7f0c767fc640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 0x7f0c6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.075+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f0c7008a650 con 0x7f0c800b1300 2024-03-24T10:36:11.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.075+0000 7f0c85638640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 0x7f0c6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.075+0000 7f0c85638640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 0x7f0c6406dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c036000 tx=0x7f0c7c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.079+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0c70055100 con 0x7f0c800b1300 2024-03-24T10:36:11.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.260+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0c800d3210 con 0x7f0c800b1300 2024-03-24T10:36:11.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.260+0000 7f0c767fc640 1 -- 172.21.15.1:0/535807572 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7f0c700587c0 con 0x7f0c800b1300 2024-03-24T10:36:11.261 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:11.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 msgr2=0x7f0c6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 0x7f0c6406dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c036000 tx=0x7f0c7c008030 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 msgr2=0x7f0c80150740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000d5f0 tx=0x7f0c7000dab0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 shutdown_connections 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0c6406b810 0x7f0c6406dcd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c8015e390 0x7f0c801f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0c800b2ed0 0x7f0c800b9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 --2- 172.21.15.1:0/535807572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c800b1300 0x7f0c80150740 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 >> 172.21.15.1:0/535807572 conn(0x7f0c800d3ee0 msgr2=0x7f0c800d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 shutdown_connections 2024-03-24T10:36:11.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.262+0000 7f0c8809d640 1 -- 172.21.15.1:0/535807572 wait complete. 2024-03-24T10:36:11.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.377+0000 7f6c4cc15640 1 Processor -- start 2024-03-24T10:36:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.377+0000 7f6c4cc15640 1 -- start start 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c4cc15640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 0x7f6c480d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c4cc15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c48162610 0x7f6c480d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c4cc15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c480c7a40 con 0x7f6c481615f0 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c4cc15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6c480cfd50 con 0x7f6c48162610 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c4cc15640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f6c480b55b0 con 0x7f6c480b4e40 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c48162610 0x7f6c480d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c48162610 0x7f6c480d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37266/0 (socket says 172.21.15.1:37266) 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 -- 172.21.15.1:0/4247679196 learned_addr learned my addr 172.21.15.1:0/4247679196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/4247679196 >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 0x7f6c480d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 -- 172.21.15.1:0/4247679196 >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 msgr2=0x7f6c480d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 --2- 172.21.15.1:0/4247679196 >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 0x7f6c480d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 -- 172.21.15.1:0/4247679196 >> v1:172.21.15.3:6789/0 conn(0x7f6c481615f0 legacy=0x7f6c48161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c46d9d640 1 -- 172.21.15.1:0/4247679196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c480b52d0 con 0x7f6c48162610 2024-03-24T10:36:11.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.378+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/4247679196 >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 0x7f6c480d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.379+0000 7f6c46d9d640 1 --2- 172.21.15.1:0/4247679196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c48162610 0x7f6c480d1c70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6c480737b0 tx=0x7f6c4002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de0cff2ad66dc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.379+0000 7f6c4559a640 1 -- 172.21.15.1:0/4247679196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4002cb20 con 0x7f6c48162610 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.379+0000 7f6c4559a640 1 -- 172.21.15.1:0/4247679196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c4002cc80 con 0x7f6c48162610 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.379+0000 7f6c4559a640 1 -- 172.21.15.1:0/4247679196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c400343f0 con 0x7f6c48162610 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 -- 172.21.15.1:0/4247679196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c48162610 msgr2=0x7f6c480d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/4247679196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c48162610 0x7f6c480d1c70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6c480737b0 tx=0x7f6c4002bf80 comp rx=0 tx=0).stop 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 -- 172.21.15.1:0/4247679196 shutdown_connections 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/4247679196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c48162610 0x7f6c480d1c70 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/4247679196 >> v2:172.21.15.3:3300/0 conn(0x7f6c480b4e40 0x7f6c480d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 -- 172.21.15.1:0/4247679196 >> 172.21.15.1:0/4247679196 conn(0x7f6c480d3ef0 msgr2=0x7f6c480d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 -- 172.21.15.1:0/4247679196 shutdown_connections 2024-03-24T10:36:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.380+0000 7f6c4cc15640 1 -- 172.21.15.1:0/4247679196 wait complete. 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.381+0000 7f6c4cc15640 1 Processor -- start 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.381+0000 7f6c4cc15640 1 -- start start 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 0x7f6c480b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c480b8d20 0x7f6c4815fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6c48153c50 con 0x7f6c481615f0 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f6c480cd2e0 con 0x7f6c480b8d20 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4cc15640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f6c480cb0f0 con 0x7f6c48162610 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37274/0 (socket says 172.21.15.1:37274) 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 -- 172.21.15.1:0/1730065373 learned_addr learned my addr 172.21.15.1:0/1730065373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c46d9d640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c480b8d20 0x7f6c4815fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 0x7f6c480b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 -- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 msgr2=0x7f6c480b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 0x7f6c480b87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 -- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c480b8d20 msgr2=0x7f6c4815fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c480b8d20 0x7f6c4815fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c4659c640 1 -- 172.21.15.1:0/1730065373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c40009b30 con 0x7f6c481615f0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.382+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 0x7f6c480b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c4659c640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800c430 tx=0x7f6c3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c380052d0 con 0x7f6c481615f0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c38005450 con 0x7f6c481615f0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c480cd510 con 0x7f6c481615f0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38007260 con 0x7f6c481615f0 2024-03-24T10:36:11.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.383+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c480cdad0 con 0x7f6c481615f0 2024-03-24T10:36:11.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.385+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4804f470 con 0x7f6c481615f0 2024-03-24T10:36:11.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.385+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6c380073c0 con 0x7f6c481615f0 2024-03-24T10:36:11.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.385+0000 7f6c3effd640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.385+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f6c380894a0 con 0x7f6c481615f0 2024-03-24T10:36:11.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.385+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.386+0000 7f6c45d9b640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 0x7f6c2806dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6c48161350 tx=0x7f6c30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.390+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6c38054000 con 0x7f6c481615f0 2024-03-24T10:36:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.645+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6c4808d640 con 0x7f6c481615f0 2024-03-24T10:36:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.646+0000 7f6c3effd640 1 -- 172.21.15.1:0/1730065373 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7f6c380576c0 con 0x7f6c481615f0 2024-03-24T10:36:11.646 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 msgr2=0x7f6c2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 0x7f6c2806dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6c48161350 tx=0x7f6c30008040 comp rx=0 tx=0).stop 2024-03-24T10:36:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 msgr2=0x7f6c481f3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800c430 tx=0x7f6c3800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 shutdown_connections 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c480b8d20 0x7f6c4815fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f6c48162610 0x7f6c480b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 --2- 172.21.15.1:0/1730065373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c481615f0 0x7f6c481f3550 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 >> 172.21.15.1:0/1730065373 conn(0x7f6c480d3ef0 msgr2=0x7f6c4815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 shutdown_connections 2024-03-24T10:36:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.648+0000 7f6c4cc15640 1 -- 172.21.15.1:0/1730065373 wait complete. 2024-03-24T10:36:11.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:11.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.764+0000 7f4fdb40b640 1 Processor -- start 2024-03-24T10:36:11.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.764+0000 7f4fdb40b640 1 -- start start 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.765+0000 7f4fdb40b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fd406fba0 0x7f4fd415e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fdb40b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd415f7b0 con 0x7f4fd406f060 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fdb40b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd415ef10 con 0x7f4fd40af9c0 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fdb40b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4fd406ff80 con 0x7f4fd406fba0 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd91a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fd406fba0 0x7f4fd415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd99a8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4fd406f060 0x7f4fd415b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37452/0 (socket says 172.21.15.1:37452) 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd99a8640 1 -- 172.21.15.1:0/3419934754 learned_addr learned my addr 172.21.15.1:0/3419934754 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd91a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4fd406fba0 0x7f4fd415e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37278/0 (socket says 172.21.15.1:37278) 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3419934754 >> v1:172.21.15.3:6789/0 conn(0x7f4fd40af9c0 legacy=0x7f4fd406e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3419934754 >> v1:172.21.15.1:6789/0 conn(0x7f4fd406f060 legacy=0x7f4fd415b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fcbfff640 1 -- 172.21.15.1:0/3419934754 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028666092 0 0) 0x7f4fd415f7b0 con 0x7f4fd406f060 2024-03-24T10:36:11.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.766+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3419934754 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd415f1f0 con 0x7f4fd406fba0 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.767+0000 7f4fd91a7640 1 --2- 172.21.15.1:0/3419934754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e930 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00097a0 tx=0x7f4fd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc19553955f2e3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.767+0000 7f4fcbfff640 1 -- 172.21.15.1:0/3419934754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0031830 con 0x7f4fd406fba0 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.767+0000 7f4fcbfff640 1 -- 172.21.15.1:0/3419934754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd0031990 con 0x7f4fd406fba0 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.767+0000 7f4fcbfff640 1 -- 172.21.15.1:0/3419934754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0031d20 con 0x7f4fd406fba0 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3419934754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 msgr2=0x7f4fd415e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3419934754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e930 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00097a0 tx=0x7f4fd002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3419934754 shutdown_connections 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3419934754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e930 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3419934754 >> 172.21.15.1:0/3419934754 conn(0x7f4fd40cd130 msgr2=0x7f4fd40cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3419934754 shutdown_connections 2024-03-24T10:36:11.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.768+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3419934754 wait complete. 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.769+0000 7f4fdb40b640 1 Processor -- start 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.769+0000 7f4fdb40b640 1 -- start start 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fd40af9c0 0x7f4fd414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fd415d940 0x7f4fd40b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd415e920 con 0x7f4fd406fba0 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd415e680 con 0x7f4fd40af9c0 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fdb40b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd4151e70 con 0x7f4fd415d940 2024-03-24T10:36:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37280/0 (socket says 172.21.15.1:37280) 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3771644099 learned_addr learned my addr 172.21.15.1:0/3771644099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fd415d940 msgr2=0x7f4fd40b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fd415d940 0x7f4fd40b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fd40af9c0 msgr2=0x7f4fd414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fd40af9c0 0x7f4fd414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.770+0000 7f4fd91a7640 1 -- 172.21.15.1:0/3771644099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd0009400 con 0x7f4fd406fba0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fd91a7640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00098c0 tx=0x7f4fd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd003ba70 con 0x7f4fd406fba0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd003bbd0 con 0x7f4fd406fba0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd4152150 con 0x7f4fd406fba0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00453f0 con 0x7f4fd406fba0 2024-03-24T10:36:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.771+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd4152710 con 0x7f4fd406fba0 2024-03-24T10:36:11.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.772+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd40afe30 con 0x7f4fd406fba0 2024-03-24T10:36:11.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.772+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4fd004e450 con 0x7f4fd406fba0 2024-03-24T10:36:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.773+0000 7f4fc9ffb640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 0x7f4fb806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.773+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f4fd00ac440 con 0x7f4fd406fba0 2024-03-24T10:36:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.773+0000 7f4fd89a6640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 0x7f4fb806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.773+0000 7f4fd89a6640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 0x7f4fb806dd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:11.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.777+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4fd40afe30 con 0x7f4fd406fba0 2024-03-24T10:36:11.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.957+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4fd40afe30 con 0x7f4fd406fba0 2024-03-24T10:36:11.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.958+0000 7f4fc9ffb640 1 -- 172.21.15.1:0/3771644099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7f4fd40afe30 con 0x7f4fd406fba0 2024-03-24T10:36:11.958 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 msgr2=0x7f4fb806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 0x7f4fb806dd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4006bc0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 msgr2=0x7f4fd415e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00098c0 tx=0x7f4fd00046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 shutdown_connections 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4fb806b840 0x7f4fb806dd00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4fd415d940 0x7f4fd40b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4fd40af9c0 0x7f4fd414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 --2- 172.21.15.1:0/3771644099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fd406fba0 0x7f4fd415e140 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 >> 172.21.15.1:0/3771644099 conn(0x7f4fd40cd130 msgr2=0x7f4fd406b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 shutdown_connections 2024-03-24T10:36:11.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:11.960+0000 7f4fdb40b640 1 -- 172.21.15.1:0/3771644099 wait complete. 2024-03-24T10:36:11.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:36:12.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.076+0000 7f4c0cd5c640 1 Processor -- start 2024-03-24T10:36:12.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.076+0000 7f4c0cd5c640 1 -- start start 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c0815e390 0x7f4c080d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 0x7f4c080d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c080b3320 0x7f4c080cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4c080b3190 con 0x7f4c0815e390 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4c0815e770 con 0x7f4c080d1000 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0cd5c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4c080d4e40 con 0x7f4c080b3320 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c0815e390 0x7f4c080d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c0815e390 0x7f4c080d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37294/0 (socket says 172.21.15.1:37294) 2024-03-24T10:36:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 -- 172.21.15.1:0/1061095823 learned_addr learned my addr 172.21.15.1:0/1061095823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c05d9b640 1 --2- 172.21.15.1:0/1061095823 >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 0x7f4c080d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c06d9d640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c080b3320 0x7f4c080cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 -- 172.21.15.1:0/1061095823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c080b3320 msgr2=0x7f4c080cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c080b3320 0x7f4c080cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 -- 172.21.15.1:0/1061095823 >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 msgr2=0x7f4c080d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1061095823 >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 0x7f4c080d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c0659c640 1 -- 172.21.15.1:0/1061095823 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c080d1e40 con 0x7f4c0815e390 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.077+0000 7f4c05d9b640 1 --2- 172.21.15.1:0/1061095823 >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 0x7f4c080d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.078+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c0815e390 0x7f4c080d24a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4c0806b4e0 tx=0x7f4bf802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244538b9e809fa9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.078+0000 7f4c0559a640 1 -- 172.21.15.1:0/1061095823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8031750 con 0x7f4c0815e390 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.078+0000 7f4c0559a640 1 -- 172.21.15.1:0/1061095823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf80318b0 con 0x7f4c0815e390 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.078+0000 7f4c0559a640 1 -- 172.21.15.1:0/1061095823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8031c40 con 0x7f4c0815e390 2024-03-24T10:36:12.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1061095823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c0815e390 msgr2=0x7f4c080d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c0815e390 0x7f4c080d24a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4c0806b4e0 tx=0x7f4bf802b7f0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1061095823 shutdown_connections 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c080b3320 0x7f4c080cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1061095823 >> v2:172.21.15.3:3300/0 conn(0x7f4c080d1000 0x7f4c080d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1061095823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c0815e390 0x7f4c080d24a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1061095823 >> 172.21.15.1:0/1061095823 conn(0x7f4c080d3ec0 msgr2=0x7f4c080d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1061095823 shutdown_connections 2024-03-24T10:36:12.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.079+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1061095823 wait complete. 2024-03-24T10:36:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.080+0000 7f4c0cd5c640 1 Processor -- start 2024-03-24T10:36:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.080+0000 7f4c0cd5c640 1 -- start start 2024-03-24T10:36:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c080d1000 0x7f4c08155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c0815e390 0x7f4c0815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c080cc0c0 con 0x7f4c080b3320 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4c080cc3a0 con 0x7f4c080d1000 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0cd5c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4c080a14a0 con 0x7f4c0815e390 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37296/0 (socket says 172.21.15.1:37296) 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 -- 172.21.15.1:0/1469065682 learned_addr learned my addr 172.21.15.1:0/1469065682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c05d9b640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c080d1000 0x7f4c08155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 -- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c0815e390 msgr2=0x7f4c0815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c0815e390 0x7f4c0815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 -- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c080d1000 msgr2=0x7f4c08155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c080d1000 0x7f4c08155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.081+0000 7f4c0659c640 1 -- 172.21.15.1:0/1469065682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf80094d0 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4c0659c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009960 tx=0x7f4bf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8033c80 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf8033e00 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c080a1780 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf803c640 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.082+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c080a1190 con 0x7f4c080b3320 2024-03-24T10:36:12.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.083+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0804f4e0 con 0x7f4c080b3320 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.083+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4bf803a040 con 0x7f4c080b3320 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.084+0000 7f4bfeffd640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 0x7f4be806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.084+0000 7f4c05d9b640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 0x7f4be806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.084+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f4bf80abed0 con 0x7f4c080b3320 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.084+0000 7f4c05d9b640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 0x7f4be806dd00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c080b1a90 tx=0x7f4bf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.088+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4bf80768f0 con 0x7f4c080b3320 2024-03-24T10:36:12.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.281+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c080d2a40 con 0x7f4c080b3320 2024-03-24T10:36:12.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.281+0000 7f4bfeffd640 1 -- 172.21.15.1:0/1469065682 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) ==== 80+0+0 (secure 0 0 0) 0x7f4bf8038080 con 0x7f4c080b3320 2024-03-24T10:36:12.282 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:36:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 msgr2=0x7f4be806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 0x7f4be806dd00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c080b1a90 tx=0x7f4bf0008040 comp rx=0 tx=0).stop 2024-03-24T10:36:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 msgr2=0x7f4c080a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009960 tx=0x7f4bf80046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 shutdown_connections 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4be806b840 0x7f4be806dd00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4c0815e390 0x7f4c0815bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c080d1000 0x7f4c08155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 --2- 172.21.15.1:0/1469065682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c080b3320 0x7f4c080a0c50 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 >> 172.21.15.1:0/1469065682 conn(0x7f4c080d3ec0 msgr2=0x7f4c080b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 shutdown_connections 2024-03-24T10:36:12.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:12.283+0000 7f4c0cd5c640 1 -- 172.21.15.1:0/1469065682 wait complete. 2024-03-24T10:36:12.294 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:36:12.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:12.294 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:12.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[58691]: fuse finished with error 0 and tester_r 0 2024-03-24T10:36:12.325 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:18.427 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:36:18.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:18.427 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:36:18.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T10:36:18.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.590+0000 7fe33cab5640 1 Processor -- start 2024-03-24T10:36:18.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.590+0000 7fe33cab5640 1 -- start start 2024-03-24T10:36:18.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.591+0000 7fe33cab5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3380d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.591+0000 7fe33cab5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe338070b90 0x7fe33806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.591+0000 7fe33cab5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3380cdd60 con 0x7fe3380b2aa0 2024-03-24T10:36:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.591+0000 7fe33cab5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fe33806eaa0 con 0x7fe338070b90 2024-03-24T10:36:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.591+0000 7fe33cab5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe338071010 con 0x7fe338144770 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe335d9b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe338070b90 0x7fe33806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe33659c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3380d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe336d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe3380b2aa0 0x7fe3380ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:54680/0 (socket says 172.21.15.1:54680) 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe336d9d640 1 -- 172.21.15.1:0/166264260 learned_addr learned my addr 172.21.15.1:0/166264260 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe33559a640 1 -- 172.21.15.1:0/166264260 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441123092 0 0) 0x7fe3380cdd60 con 0x7fe3380b2aa0 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe33559a640 1 -- 172.21.15.1:0/166264260 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe324003520 con 0x7fe3380b2aa0 2024-03-24T10:36:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe335d9b640 1 -- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 msgr2=0x7fe3380d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe335d9b640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3380d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe335d9b640 1 -- 172.21.15.1:0/166264260 >> v1:172.21.15.3:6789/0 conn(0x7fe3380b2aa0 legacy=0x7fe3380ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe335d9b640 1 -- 172.21.15.1:0/166264260 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33806df60 con 0x7fe338070b90 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.592+0000 7fe33659c640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3380d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.593+0000 7fe335d9b640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe33806da20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe3200097a0 tx=0x7fe32002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be35b20c75a46bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.593+0000 7fe33559a640 1 -- 172.21.15.1:0/166264260 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320031830 con 0x7fe338070b90 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.593+0000 7fe33559a640 1 -- 172.21.15.1:0/166264260 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe320031990 con 0x7fe338070b90 2024-03-24T10:36:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.593+0000 7fe33559a640 1 -- 172.21.15.1:0/166264260 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320031d00 con 0x7fe338070b90 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 -- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 msgr2=0x7fe33806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe33806da20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe3200097a0 tx=0x7fe32002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 -- 172.21.15.1:0/166264260 shutdown_connections 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe33806da20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 --2- 172.21.15.1:0/166264260 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3380d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 -- 172.21.15.1:0/166264260 >> 172.21.15.1:0/166264260 conn(0x7fe3380b17a0 msgr2=0x7fe3380b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 -- 172.21.15.1:0/166264260 shutdown_connections 2024-03-24T10:36:18.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.594+0000 7fe33cab5640 1 -- 172.21.15.1:0/166264260 wait complete. 2024-03-24T10:36:18.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.595+0000 7fe33cab5640 1 Processor -- start 2024-03-24T10:36:18.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.595+0000 7fe33cab5640 1 -- start start 2024-03-24T10:36:18.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe3380ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3381542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe33815ab20 con 0x7fe3380b2aa0 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fe3380ade90 con 0x7fe338070b90 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33cab5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fe3380ae170 con 0x7fe338144770 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37302/0 (socket says 172.21.15.1:37302) 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 -- 172.21.15.1:0/934499485 learned_addr learned my addr 172.21.15.1:0/934499485 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe33659c640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe3380ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe336d9d640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3381542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 -- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 msgr2=0x7fe3381542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3381542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 -- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 msgr2=0x7fe3380ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe3380ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.596+0000 7fe335d9b640 1 -- 172.21.15.1:0/934499485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe328009400 con 0x7fe3380b2aa0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe335d9b640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098c0 tx=0x7fe3200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32003a630 con 0x7fe3380b2aa0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32003a790 con 0x7fe3380b2aa0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe320009400 con 0x7fe3380b2aa0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32003ab40 con 0x7fe3380b2aa0 2024-03-24T10:36:18.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.597+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3380ae730 con 0x7fe3380b2aa0 2024-03-24T10:36:18.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.598+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe33804f4e0 con 0x7fe3380b2aa0 2024-03-24T10:36:18.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.598+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe32003aca0 con 0x7fe3380b2aa0 2024-03-24T10:36:18.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.599+0000 7fe32effd640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 0x7fe31806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:18.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.599+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fe3200abd20 con 0x7fe3380b2aa0 2024-03-24T10:36:18.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.599+0000 7fe33659c640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 0x7fe31806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:18.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.599+0000 7fe33659c640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 0x7fe31806dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe328009c30 tx=0x7fe328009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:18.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.603+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe320076880 con 0x7fe3380b2aa0 2024-03-24T10:36:18.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.825+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe338141960 con 0x7fe3380b2aa0 2024-03-24T10:36:18.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.826+0000 7fe32effd640 1 -- 172.21.15.1:0/934499485 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fe320079f40 con 0x7fe3380b2aa0 2024-03-24T10:36:18.826 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:36:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 msgr2=0x7fe31806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 0x7fe31806dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe328009c30 tx=0x7fe328009190 comp rx=0 tx=0).stop 2024-03-24T10:36:18.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 msgr2=0x7fe33813f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098c0 tx=0x7fe3200046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 shutdown_connections 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fe31806b810 0x7fe31806dcd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fe338144770 0x7fe3381542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3380b2aa0 0x7fe33813f4f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 --2- 172.21.15.1:0/934499485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe338070b90 0x7fe3380ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 >> 172.21.15.1:0/934499485 conn(0x7fe3380b17a0 msgr2=0x7fe338072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 shutdown_connections 2024-03-24T10:36:18.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:18.828+0000 7fe33cab5640 1 -- 172.21.15.1:0/934499485 wait complete. 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:36:18.839 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:36:18.840 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:36:18.840 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:18.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:18.840 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.855 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42562sec preferred_lft 42562sec 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:18.856 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::d0e8:faff:fe9a:bfd9/64 scope link 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 72:38:f3:5f:ec:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::7038:f3ff:fe5f:ec7f/64 scope link 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether ea:23:79:3c:04:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::e823:79ff:fe3c:416/64 scope link 2024-03-24T10:36:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:36:18.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:18.858 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:36:18.920 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-24T10:36:18.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:18.922 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-24T10:36:18.981 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-24T10:36:18.981 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:36:18.981 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:36:18.981 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:36:18.981 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:36:18.981 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:36:18.981 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-24T10:36:18.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:36:18.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:36:18.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:36:18.982 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-24T10:36:19.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:36:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:19.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.117 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:36:19.131 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:36:19.132 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:36:19.132 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:19.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-24T10:36:19.199 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T10:36:19.199 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:19.199 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:19.254 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:36:19.255 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:36:19.255 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:19.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:19.380 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:19.436 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:19.437 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:19.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.439 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:19.490 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T10:36:19.490 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T10:36:19.533 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-24T10:36:19.560 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-03-24T10:36:19.617 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-03-24T10:36:19.618 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:19.619 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:19.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.620 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-24T10:36:19.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-24T10:36:19.620+0000 7f5dd043c480 -1 init, newargv = 0x557733716fd0 newargc=15 2024-03-24T10:36:19.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[59655]: starting ceph client 2024-03-24T10:36:19.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[59655]: starting fuse 2024-03-24T10:36:19.635 INFO:teuthology.orchestra.run.smithi001.stdout:42 2024-03-24T10:36:19.635 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-24T10:36:19.636 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-24T10:36:19.637 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:36:19.761 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.59655.asok 2024-03-24T10:36:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:36:19.772 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59655.asok 2024-03-24T10:36:19.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59655.asok status 2024-03-24T10:36:19.908 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "59655", 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6333, 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-24T10:36:19.909 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6333 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1477460368 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1477460368 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6333 192.168.0.1:0/1477460368", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1477460368", 2024-03-24T10:36:19.910 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 25, 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 50, 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-24T10:36:19.911 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:19.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.917 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-24T10:36:19.917 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:19.932 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-24T10:36:19.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:19.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:19.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> stdin-killer -- python3 -c ' 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> import errno 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> try: 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> if True: 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> ret = os.read(fd, 1024) 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> else: 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> os.write(fd, b'"'"'content'"'"') 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> except OSError as e: 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> if e.errno != errno.EPERM: 2024-03-24T10:36:20.000 DEBUG:teuthology.orchestra.run.smithi001:> raise 2024-03-24T10:36:20.001 DEBUG:teuthology.orchestra.run.smithi001:> else: 2024-03-24T10:36:20.001 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("client does not check permission of data pool") 2024-03-24T10:36:20.001 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-24T10:36:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:Traceback (most recent call last): 2024-03-24T10:36:20.126 INFO:teuthology.orchestra.run.smithi001.stderr: File "", line 15, in 2024-03-24T10:36:20.126 INFO:teuthology.orchestra.run.smithi001.stderr:RuntimeError: client does not check permission of data pool 2024-03-24T10:36:20.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-24T10:36:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:36:20.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-24T10:36:20.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.284+0000 7f43c1193640 1 Processor -- start 2024-03-24T10:36:20.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.284+0000 7f43c1193640 1 -- start start 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43c1193640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f43bc1615f0 0x7f43bc0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43c1193640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43bc0c7a40 con 0x7f43bc0b4e40 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43c1193640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43bc0b55b0 con 0x7f43bc162610 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43c1193640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f43bc0cfd50 con 0x7f43bc1615f0 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43b259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43bc0b4e40 0x7f43bc161f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37460/0 (socket says 172.21.15.1:37460) 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43b259c640 1 -- 172.21.15.1:0/3858014467 learned_addr learned my addr 172.21.15.1:0/3858014467 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:20.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43bad9d640 1 --2- 172.21.15.1:0/3858014467 >> v2:172.21.15.3:3300/0 conn(0x7f43bc1615f0 0x7f43bc0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876224687 0 0) 0x7f43bc0c7a40 con 0x7f43bc0b4e40 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.285+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43a8003520 con 0x7f43bc0b4e40 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911135611 0 0) 0x7f43bc0b55b0 con 0x7f43bc162610 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43bc0c7a40 con 0x7f43bc162610 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3577529318 0 0) 0x7f43a8003520 con 0x7f43bc0b4e40 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 >> v2:172.21.15.3:3300/0 conn(0x7f43bc1615f0 msgr2=0x7f43bc0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 --2- 172.21.15.1:0/3858014467 >> v2:172.21.15.3:3300/0 conn(0x7f43bc1615f0 0x7f43bc0d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 >> v1:172.21.15.3:6789/0 conn(0x7f43bc162610 legacy=0x7f43bc0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc0b52d0 con 0x7f43bc0b4e40 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f43a4001670 con 0x7f43bc0b4e40 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 >> v1:172.21.15.1:6789/0 conn(0x7f43bc0b4e40 legacy=0x7f43bc161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a8003b50 0x7f43a8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 0x7f43a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43a8005180 0x7f43a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc0cfd50 con 0x7f43a80047c0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800c050 con 0x7f43a8003b50 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f43a800c210 con 0x7f43a8005180 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 0x7f43a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43bad9d640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a8003b50 0x7f43a8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43bb59e640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43a8005180 0x7f43a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 -- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43a8005180 msgr2=0x7f43a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43a8005180 0x7f43a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 -- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a8003b50 msgr2=0x7f43a8003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a8003b50 0x7f43a8003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43b259c640 1 -- 172.21.15.1:0/3858014467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a8003520 con 0x7f43a80047c0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43bb59e640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43a8005180 0x7f43a800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.286+0000 7f43bb59e640 1 -- 172.21.15.1:0/3858014467 reap_dead start 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43b259c640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 0x7f43a8004c40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f43bc08c3c0 tx=0x7f43a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da289a03d6a508a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4005ad0 con 0x7f43a80047c0 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a4005c30 con 0x7f43a80047c0 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43ba59c640 1 -- 172.21.15.1:0/3858014467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4007290 con 0x7f43a80047c0 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43c1193640 1 -- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 msgr2=0x7f43a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.287+0000 7f43c1193640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 0x7f43a8004c40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f43bc08c3c0 tx=0x7f43a80057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.288+0000 7f43c1193640 1 -- 172.21.15.1:0/3858014467 shutdown_connections 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.288+0000 7f43c1193640 1 --2- 172.21.15.1:0/3858014467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43a80047c0 0x7f43a8004c40 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.288+0000 7f43c1193640 1 -- 172.21.15.1:0/3858014467 >> 172.21.15.1:0/3858014467 conn(0x7f43bc0d3ef0 msgr2=0x7f43bc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.288+0000 7f43c1193640 1 -- 172.21.15.1:0/3858014467 shutdown_connections 2024-03-24T10:36:20.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.288+0000 7f43c1193640 1 -- 172.21.15.1:0/3858014467 wait complete. 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.289+0000 7f43c1193640 1 Processor -- start 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.289+0000 7f43c1193640 1 -- start start 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 0x7f43bc1f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 0x7f43bc1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1f54f0 con 0x7f43bc1f5070 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc0c98c0 con 0x7f43a80047c0 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43c1193640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f43bc15a570 con 0x7f43bc0c9440 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37314/0 (socket says 172.21.15.1:37314) 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 -- 172.21.15.1:0/2370432019 learned_addr learned my addr 172.21.15.1:0/2370432019 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43bb59e640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 0x7f43bc1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43bad9d640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 0x7f43bc1f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 -- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 msgr2=0x7f43bc1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 0x7f43bc1f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 -- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 msgr2=0x7f43bc1f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 0x7f43bc1f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43b259c640 1 -- 172.21.15.1:0/2370432019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc1f4b00 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43bb59e640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 0x7f43bc1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.290+0000 7f43bad9d640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 0x7f43bc1f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43b259c640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f43a402c3b0 tx=0x7f43a402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a403c040 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a4007540 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43bc1f4de0 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a402c740 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.291+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43bc0b8690 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.292+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43bc04f470 con 0x7f43bc1f5070 2024-03-24T10:36:20.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.292+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f43a4032070 con 0x7f43bc1f5070 2024-03-24T10:36:20.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.293+0000 7f43b3fff640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 0x7f439c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.293+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f43a40ab500 con 0x7f43bc1f5070 2024-03-24T10:36:20.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.293+0000 7f43bad9d640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 0x7f439c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.293+0000 7f43bad9d640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 0x7f439c06dd00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f43b4012290 tx=0x7f43b400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.297+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43a4075f20 con 0x7f43bc1f5070 2024-03-24T10:36:20.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.564+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f43bc08d640 con 0x7f43bc1f5070 2024-03-24T10:36:20.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.565+0000 7f43b3fff640 1 -- 172.21.15.1:0/2370432019 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v20) ==== 184+0+141 (secure 0 0 0) 0x7f43a40795e0 con 0x7f43bc1f5070 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBaAQBmgDRRLBAAlTn6HACbIahRP3OWvVijbw== 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stdout: caps mds = "allow" 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stdout: caps mon = "allow r" 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-24T10:36:20.566 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 msgr2=0x7f439c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 0x7f439c06dd00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f43b4012290 tx=0x7f43b400b040 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 msgr2=0x7f43bc0c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f43a402c3b0 tx=0x7f43a402e070 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 shutdown_connections 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f439c06b840 0x7f439c06dd00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f43bc0c9440 0x7f43bc1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43bc1f5070 0x7f43bc0c8f00 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 --2- 172.21.15.1:0/2370432019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43a80047c0 0x7f43bc1f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.567+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 >> 172.21.15.1:0/2370432019 conn(0x7f43bc0d3ef0 msgr2=0x7f43bc0ca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:20.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.568+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 shutdown_connections 2024-03-24T10:36:20.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.568+0000 7f43c1193640 1 -- 172.21.15.1:0/2370432019 wait complete. 2024-03-24T10:36:20.578 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T10:36:20.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:36:20.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T10:36:20.666 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T10:36:20.666 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:36:20.666 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:20.666 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:20.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.822+0000 7f238376c640 1 Processor -- start 2024-03-24T10:36:20.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.822+0000 7f238376c640 1 -- start start 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f238376c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f237c0d1000 0x7f237c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f238376c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f237c15e770 con 0x7f237c0d2bb0 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f238376c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f237c0d4e40 con 0x7f237c15e390 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f238376c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f237c0b3190 con 0x7f237c0d1000 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2380d07640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f237c0d2bb0 0x7f237c0b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54904/0 (socket says 172.21.15.1:54904) 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2380d07640 1 -- 172.21.15.1:0/3334607877 learned_addr learned my addr 172.21.15.1:0/3334607877 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786063981 0 0) 0x7f236c001110 con 0x7f237c0d2bb0 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2381d09640 1 --2- 172.21.15.1:0/3334607877 >> v2:172.21.15.3:3300/0 conn(0x7f237c0d1000 0x7f237c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2368003520 con 0x7f237c0d2bb0 2024-03-24T10:36:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 198088879 0 0) 0x7f2368003520 con 0x7f237c0d2bb0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 >> v2:172.21.15.3:3300/0 conn(0x7f237c0d1000 msgr2=0x7f237c0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 --2- 172.21.15.1:0/3334607877 >> v2:172.21.15.3:3300/0 conn(0x7f237c0d1000 0x7f237c0b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 >> v1:172.21.15.3:6789/0 conn(0x7f237c15e390 legacy=0x7f237c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.823+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c0d1e40 con 0x7f237c0d2bb0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f236c001800 con 0x7f237c0d2bb0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 >> v1:172.21.15.1:6789/0 conn(0x7f237c0d2bb0 legacy=0x7f237c0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2368003970 0x7f2368003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23680047c0 0x7f2368004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 0x7f236800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f236800c140 con 0x7f23680052a0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f236800c2b0 con 0x7f2368003970 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f236800c470 con 0x7f23680047c0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 0x7f236800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381508640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2368003970 0x7f2368003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2380d07640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23680047c0 0x7f2368004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 -- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23680047c0 msgr2=0x7f2368004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23680047c0 0x7f2368004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 -- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2368003970 msgr2=0x7f2368003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2368003970 0x7f2368003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2381d09640 1 -- 172.21.15.1:0/3334607877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2368003520 con 0x7f23680052a0 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.824+0000 7f2380d07640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f23680047c0 0x7f2368004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f2381d09640 1 -- 172.21.15.1:0/3334607877 reap_dead start 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f2381d09640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 0x7f236800b980 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2374009f90 tx=0x7f23680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101f1c6c1489cd68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237402cc20 con 0x7f23680052a0 2024-03-24T10:36:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f237400f040 con 0x7f23680052a0 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f2373fff640 1 -- 172.21.15.1:0/3334607877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374003990 con 0x7f23680052a0 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f238376c640 1 -- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 msgr2=0x7f236800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f238376c640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 0x7f236800b980 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2374009f90 tx=0x7f23680057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f238376c640 1 -- 172.21.15.1:0/3334607877 shutdown_connections 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f238376c640 1 --2- 172.21.15.1:0/3334607877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23680052a0 0x7f236800b980 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.825+0000 7f238376c640 1 -- 172.21.15.1:0/3334607877 >> 172.21.15.1:0/3334607877 conn(0x7f237c0d3ec0 msgr2=0x7f237c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.826+0000 7f238376c640 1 -- 172.21.15.1:0/3334607877 shutdown_connections 2024-03-24T10:36:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.826+0000 7f238376c640 1 -- 172.21.15.1:0/3334607877 wait complete. 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 Processor -- start 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 -- start start 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 0x7f237c1f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 0x7f237c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f237c1473d0 con 0x7f237c146a10 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f237c1477f0 con 0x7f23680052a0 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f238376c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f237c0ccb50 con 0x7f237c0cbfe0 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.827+0000 7f2381d09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58322/0 (socket says 172.21.15.1:58322) 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 -- 172.21.15.1:0/1898046702 learned_addr learned my addr 172.21.15.1:0/1898046702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2380d07640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 0x7f237c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381508640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 0x7f237c1f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 -- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 msgr2=0x7f237c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 0x7f237c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 -- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 msgr2=0x7f237c1f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 0x7f237c1f33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 -- 172.21.15.1:0/1898046702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c0cce30 con 0x7f237c146a10 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381508640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 0x7f237c1f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2380d07640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 0x7f237c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2381d09640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f237402c690 tx=0x7f2374009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237402cea0 con 0x7f237c146a10 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2374004230 con 0x7f237c146a10 2024-03-24T10:36:20.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237c0cd110 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23740045e0 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.828+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237c0cd6d0 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.829+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c0cd8f0 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.830+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2374004740 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.830+0000 7f2371ffb640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 0x7f236006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.830+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f23740aca20 con 0x7f237c146a10 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.830+0000 7f2381508640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 0x7f236006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.831+0000 7f2381508640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 0x7f236006dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2378005580 tx=0x7f2378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:20.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:20.835+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f23740774d0 con 0x7f237c146a10 2024-03-24T10:36:21.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.052+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f237c04f4e0 con 0x7f237c146a10 2024-03-24T10:36:21.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.054+0000 7f2371ffb640 1 -- 172.21.15.1:0/1898046702 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f237407ab90 con 0x7f237c146a10 2024-03-24T10:36:21.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:21.055 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 msgr2=0x7f236006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 0x7f236006dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2378005580 tx=0x7f2378009040 comp rx=0 tx=0).stop 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 msgr2=0x7f237c146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f237402c690 tx=0x7f2374009750 comp rx=0 tx=0).stop 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 shutdown_connections 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f236006b810 0x7f236006dcd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f237c146a10 0x7f237c146e90 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f237c0cbfe0 0x7f237c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 --2- 172.21.15.1:0/1898046702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23680052a0 0x7f237c1f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.056+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 >> 172.21.15.1:0/1898046702 conn(0x7f237c0d3ec0 msgr2=0x7f237c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.057+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 shutdown_connections 2024-03-24T10:36:21.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.057+0000 7f238376c640 1 -- 172.21.15.1:0/1898046702 wait complete. 2024-03-24T10:36:21.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:21.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.172+0000 7fa4f3240640 1 Processor -- start 2024-03-24T10:36:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.172+0000 7fa4f3240640 1 -- start start 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.173+0000 7fa4f3240640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4ec15e390 0x7fa4ec0d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.173+0000 7fa4f3240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 0x7fa4ec0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.173+0000 7fa4f3240640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0b0b70 con 0x7fa4ec0b2ed0 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.173+0000 7fa4f3240640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa4ec0d1d60 con 0x7fa4ec15e390 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.173+0000 7fa4f3240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ec0b0de0 con 0x7fa4ec0cfae0 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4ec15e390 0x7fa4ec0d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f17dd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 0x7fa4ec0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4ec15e390 0x7fa4ec0d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:39692/0 (socket says 172.21.15.1:39692) 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4ebfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa4ec0b2ed0 0x7fa4ec0b1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:39036/0 (socket says 172.21.15.1:39036) 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 -- 172.21.15.1:0/1852924091 learned_addr learned my addr 172.21.15.1:0/1852924091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f17dd640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 0x7fa4ec0d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:60086/0 (socket says 172.21.15.1:60086) 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 -- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 msgr2=0x7fa4ec0d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 0x7fa4ec0d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 -- 172.21.15.1:0/1852924091 >> v1:172.21.15.3:6789/0 conn(0x7fa4ec0b2ed0 legacy=0x7fa4ec0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4f0fdc640 1 -- 172.21.15.1:0/1852924091 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec15e770 con 0x7fa4ec15e390 2024-03-24T10:36:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.174+0000 7fa4eb7fe640 1 -- 172.21.15.1:0/1852924091 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797982177 0 0) 0x7fa4ec0b0b70 con 0x7fa4ec0b2ed0 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4f0fdc640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0d2310 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009e50 tx=0x7fa4e402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a4a9c3307ba76b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4eb7fe640 1 -- 172.21.15.1:0/1852924091 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e402cb00 con 0x7fa4ec15e390 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4eb7fe640 1 -- 172.21.15.1:0/1852924091 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e402cc60 con 0x7fa4ec15e390 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4eb7fe640 1 -- 172.21.15.1:0/1852924091 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40343f0 con 0x7fa4ec15e390 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4f3240640 1 -- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 msgr2=0x7fa4ec0d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4f3240640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0d2310 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009e50 tx=0x7fa4e402bf60 comp rx=0 tx=0).stop 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4f3240640 1 -- 172.21.15.1:0/1852924091 shutdown_connections 2024-03-24T10:36:21.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.175+0000 7fa4f3240640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0cfae0 0x7fa4ec0d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.176+0000 7fa4f3240640 1 --2- 172.21.15.1:0/1852924091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0d2310 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.176+0000 7fa4f3240640 1 -- 172.21.15.1:0/1852924091 >> 172.21.15.1:0/1852924091 conn(0x7fa4ec0d3ee0 msgr2=0x7fa4ec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.176+0000 7fa4f3240640 1 -- 172.21.15.1:0/1852924091 shutdown_connections 2024-03-24T10:36:21.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.176+0000 7fa4f3240640 1 -- 172.21.15.1:0/1852924091 wait complete. 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.177+0000 7fa4f3240640 1 Processor -- start 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.177+0000 7fa4f3240640 1 -- start start 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.177+0000 7fa4f3240640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 0x7fa4ec06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.177+0000 7fa4f3240640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f3240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f3240640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1f4b00 con 0x7fa4ec0cfae0 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f3240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1f4820 con 0x7fa4ec15e390 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f3240640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ec160590 con 0x7fa4ec0b2ed0 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4ebfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58324/0 (socket says 172.21.15.1:58324) 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4ebfff640 1 -- 172.21.15.1:0/3207818329 learned_addr learned my addr 172.21.15.1:0/3207818329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f0fdc640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 0x7fa4ec06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 -- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 msgr2=0x7fa4ec06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 0x7fa4ec06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 -- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 msgr2=0x7fa4ec161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 -- 172.21.15.1:0/3207818329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e4009b30 con 0x7fa4ec15e390 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4ebfff640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f0fdc640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 0x7fa4ec06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4f17dd640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0c79b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400e7f0 tx=0x7fa4d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.178+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4009590 con 0x7fa4ec15e390 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.179+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d400c940 con 0x7fa4ec15e390 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.179+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec160870 con 0x7fa4ec15e390 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.179+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d400ccd0 con 0x7fa4ec15e390 2024-03-24T10:36:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.179+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ec06c7e0 con 0x7fa4ec15e390 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.179+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ec160d90 con 0x7fa4ec15e390 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.180+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa4d4013470 con 0x7fa4ec15e390 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.180+0000 7fa4e97fa640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 0x7fa4cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.180+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fa4d4089eb0 con 0x7fa4ec15e390 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.180+0000 7fa4f0fdc640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 0x7fa4cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.181+0000 7fa4f0fdc640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 0x7fa4cc06dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4035000 tx=0x7fa4e4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.185+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa4ec160d90 con 0x7fa4ec15e390 2024-03-24T10:36:21.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.381+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4ec160d90 con 0x7fa4ec15e390 2024-03-24T10:36:21.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.382+0000 7fa4e97fa640 1 -- 172.21.15.1:0/3207818329 <== mon.1 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7fa4ec160d90 con 0x7fa4ec15e390 2024-03-24T10:36:21.382 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:21.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 msgr2=0x7fa4cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 0x7fa4cc06dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4035000 tx=0x7fa4e4030000 comp rx=0 tx=0).stop 2024-03-24T10:36:21.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 msgr2=0x7fa4ec0c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0c79b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400e7f0 tx=0x7fa4d400ecb0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 shutdown_connections 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4cc06b810 0x7fa4cc06dcd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec15e390 0x7fa4ec0c79b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4ec0cfae0 0x7fa4ec161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 --2- 172.21.15.1:0/3207818329 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4ec0b2ed0 0x7fa4ec06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 >> 172.21.15.1:0/3207818329 conn(0x7fa4ec0d3ee0 msgr2=0x7fa4ec0d1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 shutdown_connections 2024-03-24T10:36:21.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.384+0000 7fa4f3240640 1 -- 172.21.15.1:0/3207818329 wait complete. 2024-03-24T10:36:21.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.503+0000 7f41c2bb6640 1 Processor -- start 2024-03-24T10:36:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 -- start start 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41bc1615f0 0x7f41bc0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f41bc0b4e40 0x7f41bc161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc1619d0 0x7f41bc162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f41bc0cfd50 con 0x7f41bc1615f0 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f41bc0c7a40 con 0x7f41bc0b4e40 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c2bb6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc0b55b0 con 0x7f41bc1619d0 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c0952640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41bc1615f0 0x7f41bc0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c0952640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41bc1615f0 0x7f41bc0d4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58332/0 (socket says 172.21.15.1:58332) 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.504+0000 7f41c0952640 1 -- 172.21.15.1:0/455363497 learned_addr learned my addr 172.21.15.1:0/455363497 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41bbfff640 1 --2- 172.21.15.1:0/455363497 >> v2:172.21.15.3:3300/0 conn(0x7f41bc0b4e40 0x7f41bc161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c1153640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc1619d0 0x7f41bc162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 -- 172.21.15.1:0/455363497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc1619d0 msgr2=0x7f41bc162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc1619d0 0x7f41bc162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 -- 172.21.15.1:0/455363497 >> v2:172.21.15.3:3300/0 conn(0x7f41bc0b4e40 msgr2=0x7f41bc161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 --2- 172.21.15.1:0/455363497 >> v2:172.21.15.3:3300/0 conn(0x7f41bc0b4e40 0x7f41bc161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 -- 172.21.15.1:0/455363497 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc0b52d0 con 0x7f41bc1615f0 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.505+0000 7f41c0952640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc0d4a10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0737b0 tx=0x7f41b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a01336121cfb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41bb7fe640 1 -- 172.21.15.1:0/455363497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031640 con 0x7f41bc1615f0 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41bb7fe640 1 -- 172.21.15.1:0/455363497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b00317a0 con 0x7f41bc1615f0 2024-03-24T10:36:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41bb7fe640 1 -- 172.21.15.1:0/455363497 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031b30 con 0x7f41bc1615f0 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 -- 172.21.15.1:0/455363497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 msgr2=0x7f41bc0d4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc0d4a10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0737b0 tx=0x7f41b002b8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 -- 172.21.15.1:0/455363497 shutdown_connections 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc1619d0 0x7f41bc162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/455363497 >> v2:172.21.15.3:3300/0 conn(0x7f41bc0b4e40 0x7f41bc161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/455363497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc0d4a10 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 -- 172.21.15.1:0/455363497 >> 172.21.15.1:0/455363497 conn(0x7f41bc0d3ef0 msgr2=0x7f41bc0d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 -- 172.21.15.1:0/455363497 shutdown_connections 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.506+0000 7f41c2bb6640 1 -- 172.21.15.1:0/455363497 wait complete. 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.507+0000 7f41c2bb6640 1 Processor -- start 2024-03-24T10:36:21.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 -- start start 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41bc1619d0 0x7f41bc0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc0b8cf0 0x7f41bc0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc0cc240 con 0x7f41bc1615f0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc1560f0 con 0x7f41bc1619d0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c2bb6640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc0cb6c0 con 0x7f41bc0b8cf0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.508+0000 7f41c0952640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58346/0 (socket says 172.21.15.1:58346) 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 -- 172.21.15.1:0/1116969541 learned_addr learned my addr 172.21.15.1:0/1116969541 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41bbfff640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41bc1619d0 0x7f41bc0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c1153640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc0b8cf0 0x7f41bc0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 -- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc0b8cf0 msgr2=0x7f41bc0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc0b8cf0 0x7f41bc0c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 -- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41bc1619d0 msgr2=0x7f41bc0cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41bc1619d0 0x7f41bc0cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 -- 172.21.15.1:0/1116969541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00094d0 con 0x7f41bc1615f0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c0952640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b003a8b0 con 0x7f41bc1615f0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b003aa10 con 0x7f41bc1615f0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b003adc0 con 0x7f41bc1615f0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc0cb9a0 con 0x7f41bc1615f0 2024-03-24T10:36:21.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.509+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc0cbf60 con 0x7f41bc1615f0 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.510+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41bc04f470 con 0x7f41bc1615f0 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.511+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f41b002f050 con 0x7f41bc1615f0 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.511+0000 7f41b97fa640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 0x7f41a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.511+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f41b00ac060 con 0x7f41bc1615f0 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.511+0000 7f41bbfff640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 0x7f41a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.512+0000 7f41bbfff640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 0x7f41a006dcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0c9de0 tx=0x7f41a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.516+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f41b0076b10 con 0x7f41bc1615f0 2024-03-24T10:36:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.696+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41bc0d3090 con 0x7f41bc1615f0 2024-03-24T10:36:21.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.696+0000 7f41b97fa640 1 -- 172.21.15.1:0/1116969541 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7f41b007a1d0 con 0x7f41bc1615f0 2024-03-24T10:36:21.696 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 msgr2=0x7f41a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 0x7f41a006dcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0c9de0 tx=0x7f41a8008040 comp rx=0 tx=0).stop 2024-03-24T10:36:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 msgr2=0x7f41bc155250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f41b0000c00 tx=0x7f41b00046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 shutdown_connections 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f41a006b810 0x7f41a006dcd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f41bc0b8cf0 0x7f41bc0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41bc1619d0 0x7f41bc0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 --2- 172.21.15.1:0/1116969541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41bc1615f0 0x7f41bc155250 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 >> 172.21.15.1:0/1116969541 conn(0x7f41bc0d3ef0 msgr2=0x7f41bc06c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 shutdown_connections 2024-03-24T10:36:21.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.698+0000 7f41c2bb6640 1 -- 172.21.15.1:0/1116969541 wait complete. 2024-03-24T10:36:21.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:21.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.817+0000 7fec18f84640 1 Processor -- start 2024-03-24T10:36:21.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 -- start start 2024-03-24T10:36:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec1406fe90 0x7fec1406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec140ccd00 0x7fec140cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec140cc2b0 con 0x7fec1415e2c0 2024-03-24T10:36:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fec140cc870 con 0x7fec1406fe90 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.818+0000 7fec18f84640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fec140d0d50 con 0x7fec140ccd00 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec1406fe90 0x7fec1406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec1406fe90 0x7fec1406f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58356/0 (socket says 172.21.15.1:58356) 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec1259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec1415e2c0 0x7fec1406f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54908/0 (socket says 172.21.15.1:54908) 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 -- 172.21.15.1:0/2296658139 learned_addr learned my addr 172.21.15.1:0/2296658139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec12d9d640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec140ccd00 0x7fec140cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 -- 172.21.15.1:0/2296658139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec140ccd00 msgr2=0x7fec140cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec140ccd00 0x7fec140cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 -- 172.21.15.1:0/2296658139 >> v1:172.21.15.1:6789/0 conn(0x7fec1415e2c0 legacy=0x7fec1406f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec11d9b640 1 -- 172.21.15.1:0/2296658139 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec140d1030 con 0x7fec1406fe90 2024-03-24T10:36:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.819+0000 7fec1159a640 1 -- 172.21.15.1:0/2296658139 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287815905 0 0) 0x7fec140cc2b0 con 0x7fec1415e2c0 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec11d9b640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1406fe90 0x7fec1406f130 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7febfc009870 tx=0x7febfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a06827e7aaa2911 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec1159a640 1 -- 172.21.15.1:0/2296658139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc031640 con 0x7fec1406fe90 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec1159a640 1 -- 172.21.15.1:0/2296658139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc0317a0 con 0x7fec1406fe90 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec1159a640 1 -- 172.21.15.1:0/2296658139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc031b30 con 0x7fec1406fe90 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec18f84640 1 -- 172.21.15.1:0/2296658139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1406fe90 msgr2=0x7fec1406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec18f84640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1406fe90 0x7fec1406f130 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7febfc009870 tx=0x7febfc02b8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec18f84640 1 -- 172.21.15.1:0/2296658139 shutdown_connections 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.820+0000 7fec18f84640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec140ccd00 0x7fec140cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.821+0000 7fec18f84640 1 --2- 172.21.15.1:0/2296658139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec1406fe90 0x7fec1406f130 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.821+0000 7fec18f84640 1 -- 172.21.15.1:0/2296658139 >> 172.21.15.1:0/2296658139 conn(0x7fec140d1c60 msgr2=0x7fec140d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.821+0000 7fec18f84640 1 -- 172.21.15.1:0/2296658139 shutdown_connections 2024-03-24T10:36:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.821+0000 7fec18f84640 1 -- 172.21.15.1:0/2296658139 wait complete. 2024-03-24T10:36:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.822+0000 7fec18f84640 1 Processor -- start 2024-03-24T10:36:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.822+0000 7fec18f84640 1 -- start start 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.822+0000 7fec18f84640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec1406fe90 0x7fec14157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 0x7fec1415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec141579b0 con 0x7fec140ccd00 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fec141f7da0 con 0x7fec1415e2c0 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fec140b4770 con 0x7fec1406fe90 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58358/0 (socket says 172.21.15.1:58358) 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 -- 172.21.15.1:0/3917277408 learned_addr learned my addr 172.21.15.1:0/3917277408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec12d9d640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 0x7fec1415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec1259c640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec1406fe90 0x7fec14157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 -- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec1406fe90 msgr2=0x7fec14157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec1406fe90 0x7fec14157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 -- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 msgr2=0x7fec1415fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 0x7fec1415fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 -- 172.21.15.1:0/3917277408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc0094d0 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec12d9d640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 0x7fec1415fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec11d9b640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7febfc000c00 tx=0x7febfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc03ac60 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc03adc0 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec141f8230 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc044630 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.823+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec141f7750 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.825+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febfc02f050 con 0x7fec140ccd00 2024-03-24T10:36:21.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.825+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec14070a70 con 0x7fec140ccd00 2024-03-24T10:36:21.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.825+0000 7fec0affd640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 0x7febf406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:21.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.825+0000 7fec1259c640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 0x7febf406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:21.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.825+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7febfc0abff0 con 0x7fec140ccd00 2024-03-24T10:36:21.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.826+0000 7fec1259c640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 0x7febf406dd00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec0c000f60 tx=0x7fec0c007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:21.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:21.829+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febfc076aa0 con 0x7fec140ccd00 2024-03-24T10:36:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.009+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fec1404f4e0 con 0x7fec140ccd00 2024-03-24T10:36:22.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.010+0000 7fec0affd640 1 -- 172.21.15.1:0/3917277408 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7febfc07a160 con 0x7fec140ccd00 2024-03-24T10:36:22.010 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:22.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 msgr2=0x7febf406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 0x7febf406dd00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec0c000f60 tx=0x7fec0c007c00 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 msgr2=0x7fec140b3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7febfc000c00 tx=0x7febfc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 shutdown_connections 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7febf406b840 0x7febf406dd00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec1415e2c0 0x7fec1415fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec140ccd00 0x7fec140b3e60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 --2- 172.21.15.1:0/3917277408 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fec1406fe90 0x7fec14157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 >> 172.21.15.1:0/3917277408 conn(0x7fec140d1c60 msgr2=0x7fec1415d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 shutdown_connections 2024-03-24T10:36:22.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.012+0000 7fec18f84640 1 -- 172.21.15.1:0/3917277408 wait complete. 2024-03-24T10:36:22.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:36:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.125+0000 7f61be0d4640 1 Processor -- start 2024-03-24T10:36:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.125+0000 7f61be0d4640 1 -- start start 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61be0d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b80d1000 0x7f61b80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61be0d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b815e770 con 0x7f61b815e390 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61be0d4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b80d4e40 con 0x7f61b80d2bb0 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61be0d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f61b80b3190 con 0x7f61b80d1000 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61b815e390 0x7f61b80d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54912/0 (socket says 172.21.15.1:54912) 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b77fe640 1 -- 172.21.15.1:0/4282257436 learned_addr learned my addr 172.21.15.1:0/4282257436 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b80d1000 0x7f61b80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872401755 0 0) 0x7f61a8001110 con 0x7f61b815e390 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61a0003520 con 0x7f61b815e390 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258437011 0 0) 0x7f61b80d4e40 con 0x7f61b80d2bb0 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61a8001110 con 0x7f61b80d2bb0 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 478770002 0 0) 0x7f61a0003520 con 0x7f61b815e390 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b80d1000 msgr2=0x7f61b80b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b80d1000 0x7f61b80b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 >> v1:172.21.15.3:6789/0 conn(0x7f61b80d2bb0 legacy=0x7f61b80b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b80d1e40 con 0x7f61b815e390 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f61a8001800 con 0x7f61b815e390 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 >> v1:172.21.15.1:6789/0 conn(0x7f61b815e390 legacy=0x7f61b80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61a0003b30 0x7f61a0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61a00047c0 0x7f61a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a0005180 0x7f61a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f61a000c080 con 0x7f61a0005180 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f61a000c1f0 con 0x7f61a00047c0 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f61a000c3b0 con 0x7f61a0003b30 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.126+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b80d1000 0x7f61b80b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a0005180 0x7f61a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b77fe640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61a0003b30 0x7f61a0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b6ffd640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61a00047c0 0x7f61a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 -- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61a0003b30 msgr2=0x7f61a0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61a0003b30 0x7f61a0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 -- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61a00047c0 msgr2=0x7f61a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61a00047c0 0x7f61a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.127+0000 7f61b7fff640 1 -- 172.21.15.1:0/4282257436 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a0003520 con 0x7f61a0005180 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61b7fff640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a0005180 0x7f61a000b980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f61b80d13e0 tx=0x7f61a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd6c9ae894b39d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b002cce0 con 0x7f61a0005180 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b000f040 con 0x7f61a0005180 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61b67fc640 1 -- 172.21.15.1:0/4282257436 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0003bb0 con 0x7f61a0005180 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 -- 172.21.15.1:0/4282257436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a0005180 msgr2=0x7f61a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 --2- 172.21.15.1:0/4282257436 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61a0005180 0x7f61a000b980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f61b80d13e0 tx=0x7f61a00057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61b7fff640 1 -- 172.21.15.1:0/4282257436 reap_dead start 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 -- 172.21.15.1:0/4282257436 shutdown_connections 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 -- 172.21.15.1:0/4282257436 >> 172.21.15.1:0/4282257436 conn(0x7f61b80d3ec0 msgr2=0x7f61b80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:22.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 -- 172.21.15.1:0/4282257436 shutdown_connections 2024-03-24T10:36:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.128+0000 7f61be0d4640 1 -- 172.21.15.1:0/4282257436 wait complete. 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.129+0000 7f61be0d4640 1 Processor -- start 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 -- start start 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b806ce60 0x7f61b80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61b80b9600 0x7f61b8159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8159300 con 0x7f61b80cb4d0 2024-03-24T10:36:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8156720 con 0x7f61b80b9600 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61be0d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f61b8158470 con 0x7f61b806ce60 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61b6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61b6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58376/0 (socket says 172.21.15.1:58376) 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.130+0000 7f61b6ffd640 1 -- 172.21.15.1:0/3448889461 learned_addr learned my addr 172.21.15.1:0/3448889461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b7fff640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61b80b9600 0x7f61b8159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b77fe640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b806ce60 0x7f61b80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 -- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b806ce60 msgr2=0x7f61b80caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b806ce60 0x7f61b80caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 -- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61b80b9600 msgr2=0x7f61b8159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61b80b9600 0x7f61b8159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 -- 172.21.15.1:0/3448889461 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b81586f0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61b6ffd640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00f770 tx=0x7f61ac00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0050d0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac005270 con 0x7f61b80cb4d0 2024-03-24T10:36:22.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81589d0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00d730 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.131+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b8158f90 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.132+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b8159120 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.133+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61ac00daa0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.133+0000 7f619ffff640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 0x7f61a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.133+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f61ac08a540 con 0x7f61b80cb4d0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.133+0000 7f61b77fe640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 0x7f61a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.134+0000 7f61b77fe640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 0x7f61a406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f61a8001eb0 tx=0x7f61a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.137+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61ac054ed0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.332+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f61b804f4e0 con 0x7f61b80cb4d0 2024-03-24T10:36:22.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.332+0000 7f619ffff640 1 -- 172.21.15.1:0/3448889461 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) ==== 80+0+0 (secure 0 0 0) 0x7f61ac058590 con 0x7f61b80cb4d0 2024-03-24T10:36:22.333 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 msgr2=0x7f61a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 0x7f61a406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f61a8001eb0 tx=0x7f61a8009040 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 msgr2=0x7f61b80b90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00f770 tx=0x7f61ac00fc30 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 shutdown_connections 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f61a406b810 0x7f61a406dcd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61b80b9600 0x7f61b8159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b80cb4d0 0x7f61b80b90c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 --2- 172.21.15.1:0/3448889461 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f61b806ce60 0x7f61b80caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.334+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 >> 172.21.15.1:0/3448889461 conn(0x7f61b80d3ec0 msgr2=0x7f61b80cf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:22.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.335+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 shutdown_connections 2024-03-24T10:36:22.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.335+0000 7f61be0d4640 1 -- 172.21.15.1:0/3448889461 wait complete. 2024-03-24T10:36:22.346 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-24T10:36:22.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:22.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T10:36:22.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[59655]: fuse finished with error 0 and tester_r 0 2024-03-24T10:36:22.375 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:36:22.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:22.375 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:36:22.465 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:36:22.465 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:36:22.465 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:22.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:36:22.484 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:36:22.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:22.484 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T10:36:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:36:22.542 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T10:36:22.542 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:36:22.542 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:22.542 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:22.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:36:22.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:36:22.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:22.558 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T10:36:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:36:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T10:36:22.612 INFO:tasks.cephfs.mount:Terminating background process 2024-03-24T10:36:22.612 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:36:22.613 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:22.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:36:22.630 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:36:22.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:22.631 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T10:36:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:36:22.687 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T10:36:22.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.788+0000 7f54efced640 1 Processor -- start 2024-03-24T10:36:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.788+0000 7f54efced640 1 -- start start 2024-03-24T10:36:22.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54efced640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54e80d2bb0 0x7f54e80b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54efced640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e80b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54efced640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54e815e770 con 0x7f54e815e390 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54efced640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f54e80b3190 con 0x7f54e80d2bb0 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54efced640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f54e80d4e40 con 0x7f54e80d1000 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54e80d2bb0 0x7f54e80b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54e80d2bb0 0x7f54e80b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58378/0 (socket says 172.21.15.1:58378) 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 -- 172.21.15.1:0/2507582588 learned_addr learned my addr 172.21.15.1:0/2507582588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ee28a640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e80b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 -- 172.21.15.1:0/2507582588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 msgr2=0x7f54e80b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e80b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 -- 172.21.15.1:0/2507582588 >> v1:172.21.15.3:6789/0 conn(0x7f54e815e390 legacy=0x7f54e80d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ed288640 1 -- 172.21.15.1:0/2507582588 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e80d1e40 con 0x7f54e80d2bb0 2024-03-24T10:36:22.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.790+0000 7f54ee28a640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e80b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.791+0000 7f54ed288640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b3b90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009870 tx=0x7f54d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7d2980fc6bb546 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.791+0000 7f54eca87640 1 -- 172.21.15.1:0/2507582588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d80317c0 con 0x7f54e80d2bb0 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.791+0000 7f54eca87640 1 -- 172.21.15.1:0/2507582588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d8031920 con 0x7f54e80d2bb0 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.791+0000 7f54eca87640 1 -- 172.21.15.1:0/2507582588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8031cb0 con 0x7f54e80d2bb0 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 -- 172.21.15.1:0/2507582588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 msgr2=0x7f54e80b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b3b90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009870 tx=0x7f54d802b860 comp rx=0 tx=0).stop 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 -- 172.21.15.1:0/2507582588 shutdown_connections 2024-03-24T10:36:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e80b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 --2- 172.21.15.1:0/2507582588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b3b90 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 -- 172.21.15.1:0/2507582588 >> 172.21.15.1:0/2507582588 conn(0x7f54e80d3ec0 msgr2=0x7f54e80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 -- 172.21.15.1:0/2507582588 shutdown_connections 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.792+0000 7f54efced640 1 -- 172.21.15.1:0/2507582588 wait complete. 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.793+0000 7f54efced640 1 Processor -- start 2024-03-24T10:36:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.793+0000 7f54efced640 1 -- start start 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e809ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e815e390 0x7f54e80cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54e809f190 con 0x7f54e80d2bb0 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f54e809dbb0 con 0x7f54e815e390 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54efced640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f54e809de90 con 0x7f54e80d1000 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58392/0 (socket says 172.21.15.1:58392) 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 -- 172.21.15.1:0/2677289571 learned_addr learned my addr 172.21.15.1:0/2677289571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54eda89640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e809ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ee28a640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e815e390 0x7f54e80cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 -- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 msgr2=0x7f54e809ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e809ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 -- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e815e390 msgr2=0x7f54e80cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e815e390 0x7f54e80cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 -- 172.21.15.1:0/2677289571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d80094d0 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.794+0000 7f54ed288640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f54d802b8a0 tx=0x7f54d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.795+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8032860 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.795+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d8032a00 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.795+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e809e170 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.795+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8032db0 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.795+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e809e730 con 0x7f54e80d2bb0 2024-03-24T10:36:22.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.796+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e809e920 con 0x7f54e80d2bb0 2024-03-24T10:36:22.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.796+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f54d804e450 con 0x7f54e80d2bb0 2024-03-24T10:36:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.796+0000 7f54de7fc640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 0x7f54cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.796+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f54d80ac2a0 con 0x7f54e80d2bb0 2024-03-24T10:36:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.797+0000 7f54eda89640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 0x7f54cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:22.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.797+0000 7f54eda89640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 0x7f54cc06dcd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f54e80d3ae0 tx=0x7f54e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:22.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:22.805+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f54d8076ec0 con 0x7f54e80d2bb0 2024-03-24T10:36:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.025+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54e804f4e0 con 0x7f54e80d2bb0 2024-03-24T10:36:23.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.025+0000 7f54de7fc640 1 -- 172.21.15.1:0/2677289571 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f54d807a580 con 0x7f54e80d2bb0 2024-03-24T10:36:23.026 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:23.026 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:36:23.026 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:36:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 msgr2=0x7f54cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 0x7f54cc06dcd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f54e80d3ae0 tx=0x7f54e4009040 comp rx=0 tx=0).stop 2024-03-24T10:36:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 msgr2=0x7f54e80b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f54d802b8a0 tx=0x7f54d80046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 shutdown_connections 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f54cc06b810 0x7f54cc06dcd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54e815e390 0x7f54e80cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d2bb0 0x7f54e80b1510 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 --2- 172.21.15.1:0/2677289571 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f54e80d1000 0x7f54e809ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 >> 172.21.15.1:0/2677289571 conn(0x7f54e80d3ec0 msgr2=0x7f54e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 shutdown_connections 2024-03-24T10:36:23.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.028+0000 7f54efced640 1 -- 172.21.15.1:0/2677289571 wait complete. 2024-03-24T10:36:23.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.147+0000 7f3d02aeb640 1 Processor -- start 2024-03-24T10:36:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.147+0000 7f3d02aeb640 1 -- start start 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc15e390 0x7f3cfc0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3cfc0b3320 0x7f3cfc0cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3cfc0b3190 con 0x7f3cfc0d1000 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3cfc15e770 con 0x7f3cfc0b3320 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d02aeb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3cfc0d4e40 con 0x7f3cfc15e390 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58408/0 (socket says 172.21.15.1:58408) 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 -- 172.21.15.1:0/504009949 learned_addr learned my addr 172.21.15.1:0/504009949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d00887640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc15e390 0x7f3cfc0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 -- 172.21.15.1:0/504009949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc15e390 msgr2=0x7f3cfc0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc15e390 0x7f3cfc0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 -- 172.21.15.1:0/504009949 >> v2:172.21.15.3:3300/0 conn(0x7f3cfc0b3320 msgr2=0x7f3cfc0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3d01088640 1 --2- 172.21.15.1:0/504009949 >> v2:172.21.15.3:3300/0 conn(0x7f3cfc0b3320 0x7f3cfc0cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 --2- 172.21.15.1:0/504009949 >> v2:172.21.15.3:3300/0 conn(0x7f3cfc0b3320 0x7f3cfc0cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.148+0000 7f3cfbfff640 1 -- 172.21.15.1:0/504009949 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc0d1e40 con 0x7f3cfc0d1000 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.149+0000 7f3cfbfff640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0b5960 tx=0x7f3ce802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb920372cffc5a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.149+0000 7f3cfb7fe640 1 -- 172.21.15.1:0/504009949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8031750 con 0x7f3cfc0d1000 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.149+0000 7f3cfb7fe640 1 -- 172.21.15.1:0/504009949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce80318b0 con 0x7f3cfc0d1000 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3cfb7fe640 1 -- 172.21.15.1:0/504009949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8031c40 con 0x7f3cfc0d1000 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 -- 172.21.15.1:0/504009949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc0d1000 msgr2=0x7f3cfc0d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0b5960 tx=0x7f3ce802b7f0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 -- 172.21.15.1:0/504009949 shutdown_connections 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/504009949 >> v2:172.21.15.3:3300/0 conn(0x7f3cfc0b3320 0x7f3cfc0cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc0d1000 0x7f3cfc0d29e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/504009949 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc15e390 0x7f3cfc0d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 -- 172.21.15.1:0/504009949 >> 172.21.15.1:0/504009949 conn(0x7f3cfc0d3ec0 msgr2=0x7f3cfc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 -- 172.21.15.1:0/504009949 shutdown_connections 2024-03-24T10:36:23.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.150+0000 7f3d02aeb640 1 -- 172.21.15.1:0/504009949 wait complete. 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.151+0000 7f3d02aeb640 1 Processor -- start 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.151+0000 7f3d02aeb640 1 -- start start 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cfc0b3320 0x7f3cfc152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc0d1000 0x7f3cfc0cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc0cce90 con 0x7f3cfc15e390 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc0cd170 con 0x7f3cfc0b3320 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d02aeb640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3cfc0cd450 con 0x7f3cfc0d1000 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d01088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d01088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58414/0 (socket says 172.21.15.1:58414) 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d01088640 1 -- 172.21.15.1:0/2845786826 learned_addr learned my addr 172.21.15.1:0/2845786826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3d00887640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cfc0b3320 0x7f3cfc152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.152+0000 7f3cfbfff640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc0d1000 0x7f3cfc0cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 -- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc0d1000 msgr2=0x7f3cfc0cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc0d1000 0x7f3cfc0cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 -- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cfc0b3320 msgr2=0x7f3cfc152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cfc0b3320 0x7f3cfc152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 -- 172.21.15.1:0/2845786826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce80094d0 con 0x7f3cfc15e390 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d01088640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400cd60 tx=0x7f3cf400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4003b90 con 0x7f3cfc15e390 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf4003d10 con 0x7f3cfc15e390 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4004260 con 0x7f3cfc15e390 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cfc0cd730 con 0x7f3cfc15e390 2024-03-24T10:36:23.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.153+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cfc152750 con 0x7f3cfc15e390 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.154+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cfc0cdc50 con 0x7f3cfc15e390 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.155+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3cf40043c0 con 0x7f3cfc15e390 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.155+0000 7f3cf97fa640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 0x7f3ce006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.155+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f3cf4012070 con 0x7f3cfc15e390 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.155+0000 7f3d00887640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 0x7f3ce006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.155+0000 7f3d00887640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 0x7f3ce006dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc15d1c0 tx=0x7f3cf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.163+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3cfc0cdc50 con 0x7f3cfc15e390 2024-03-24T10:36:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.344+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cfc0cdc50 con 0x7f3cfc15e390 2024-03-24T10:36:23.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.346+0000 7f3cf97fa640 1 -- 172.21.15.1:0/2845786826 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f3cfc0cdc50 con 0x7f3cfc15e390 2024-03-24T10:36:23.346 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:23.346 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 msgr2=0x7f3ce006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 0x7f3ce006dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc15d1c0 tx=0x7f3cf0008040 comp rx=0 tx=0).stop 2024-03-24T10:36:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 msgr2=0x7f3cfc06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400cd60 tx=0x7f3cf400a660 comp rx=0 tx=0).stop 2024-03-24T10:36:23.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 shutdown_connections 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3ce006b810 0x7f3ce006dcd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3cfc15e390 0x7f3cfc06c250 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3cfc0d1000 0x7f3cfc0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 --2- 172.21.15.1:0/2845786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cfc0b3320 0x7f3cfc152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 >> 172.21.15.1:0/2845786826 conn(0x7f3cfc0d3ec0 msgr2=0x7f3cfc0b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 shutdown_connections 2024-03-24T10:36:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.348+0000 7f3d02aeb640 1 -- 172.21.15.1:0/2845786826 wait complete. 2024-03-24T10:36:23.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:42.558975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:35:49.256489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:35:49.948166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:36:23.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:36:23.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.464+0000 7f3ca2ef4640 1 Processor -- start 2024-03-24T10:36:23.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.464+0000 7f3ca2ef4640 1 -- start start 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca2ef4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca2ef4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c9c15c4c0 con 0x7f3c9c0b2e40 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca2ef4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c9c0cd5a0 con 0x7f3c9c0cfd40 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca2ef4640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3c9c0cd880 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca0c90640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca1491640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c9c0b2e40 0x7f3c9c0ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54922/0 (socket says 172.21.15.1:54922) 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca1491640 1 -- 172.21.15.1:0/964149438 learned_addr learned my addr 172.21.15.1:0/964149438 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca0c90640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58430/0 (socket says 172.21.15.1:58430) 2024-03-24T10:36:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca0c90640 1 -- 172.21.15.1:0/964149438 >> v1:172.21.15.3:6789/0 conn(0x7f3c9c0cfd40 legacy=0x7f3c9c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca0c90640 1 -- 172.21.15.1:0/964149438 >> v1:172.21.15.1:6789/0 conn(0x7f3c9c0b2e40 legacy=0x7f3c9c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3ca0c90640 1 -- 172.21.15.1:0/964149438 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c0cdb60 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.465+0000 7f3c937fe640 1 -- 172.21.15.1:0/964149438 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077356692 0 0) 0x7f3c9c15c4c0 con 0x7f3c9c0b2e40 2024-03-24T10:36:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.466+0000 7f3ca0c90640 1 --2- 172.21.15.1:0/964149438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3c980097d0 tx=0x7f3c9802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7d198cd63dddca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.466+0000 7f3c937fe640 1 -- 172.21.15.1:0/964149438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98030610 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.466+0000 7f3c937fe640 1 -- 172.21.15.1:0/964149438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c98030770 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.466+0000 7f3c937fe640 1 -- 172.21.15.1:0/964149438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98030b00 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/964149438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 msgr2=0x7f3c9c0cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/964149438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3c980097d0 tx=0x7f3c9802b960 comp rx=0 tx=0).stop 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/964149438 shutdown_connections 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/964149438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0cf800 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/964149438 >> 172.21.15.1:0/964149438 conn(0x7f3c9c0d1c30 msgr2=0x7f3c9c0d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/964149438 shutdown_connections 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.467+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/964149438 wait complete. 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.468+0000 7f3ca2ef4640 1 Processor -- start 2024-03-24T10:36:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.468+0000 7f3ca2ef4640 1 -- start start 2024-03-24T10:36:23.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3c9c0b2e40 0x7f3c9c152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c0cfd40 0x7f3c9c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1f4c90 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c06c350 con 0x7f3c9c0cfd40 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca2ef4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3c9c06c630 con 0x7f3c9c0b2e40 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca1491640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca1491640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58436/0 (socket says 172.21.15.1:58436) 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca1491640 1 -- 172.21.15.1:0/403935212 learned_addr learned my addr 172.21.15.1:0/403935212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3c93fff640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c0cfd40 0x7f3c9c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.469+0000 7f3ca0c90640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3c9c0b2e40 0x7f3c9c152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 -- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3c9c0b2e40 msgr2=0x7f3c9c152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3c9c0b2e40 0x7f3c9c152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 -- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c0cfd40 msgr2=0x7f3c9c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c0cfd40 0x7f3c9c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 -- 172.21.15.1:0/403935212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c98009400 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca1491640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c700 tx=0x7f3c9400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400ce10 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9400a640 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9c06c910 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400aa70 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.470+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9c06ced0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.471+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c9c06d0c0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.472+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c9400abd0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.472+0000 7f3c917fa640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 0x7f3c8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.472+0000 7f3ca0c90640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 0x7f3c8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.472+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f3c940898d0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.473+0000 7f3ca0c90640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 0x7f3c8806dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c98036000 tx=0x7f3c98008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.477+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c940543a0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.662+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3c9c04f4e0 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.662+0000 7f3c917fa640 1 -- 172.21.15.1:0/403935212 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7f3c94018020 con 0x7f3c9c15c0e0 2024-03-24T10:36:23.663 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 msgr2=0x7f3c8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 0x7f3c8806dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c98036000 tx=0x7f3c98008030 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 msgr2=0x7f3c9c0ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400c700 tx=0x7f3c9400cbc0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 shutdown_connections 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3c8806b810 0x7f3c8806dcd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c9c15c0e0 0x7f3c9c0ce880 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c0cfd40 0x7f3c9c0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 --2- 172.21.15.1:0/403935212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3c9c0b2e40 0x7f3c9c152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 >> 172.21.15.1:0/403935212 conn(0x7f3c9c0d1c30 msgr2=0x7f3c9c0704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 shutdown_connections 2024-03-24T10:36:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.664+0000 7f3ca2ef4640 1 -- 172.21.15.1:0/403935212 wait complete. 2024-03-24T10:36:23.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:36:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.780+0000 7f5385c88640 1 Processor -- start 2024-03-24T10:36:23.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.780+0000 7f5385c88640 1 -- start start 2024-03-24T10:36:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 0x7f53800d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53800b4e40 0x7f5380161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 0x7f5380162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f53800cfd50 con 0x7f53800b4e40 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f53800c7a40 con 0x7f53801615f0 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f5385c88640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f53800b55b0 con 0x7f53801619d0 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53800b4e40 0x7f5380161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f53800b4e40 0x7f5380161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58446/0 (socket says 172.21.15.1:58446) 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 -- 172.21.15.1:0/4137226731 learned_addr learned my addr 172.21.15.1:0/4137226731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537f7fe640 1 --2- 172.21.15.1:0/4137226731 >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 0x7f53800d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537ffff640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 0x7f5380162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 -- 172.21.15.1:0/4137226731 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 msgr2=0x7f5380162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 0x7f5380162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 -- 172.21.15.1:0/4137226731 >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 msgr2=0x7f53800d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 --2- 172.21.15.1:0/4137226731 >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 0x7f53800d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537effd640 1 -- 172.21.15.1:0/4137226731 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800b52d0 con 0x7f53800b4e40 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537f7fe640 1 --2- 172.21.15.1:0/4137226731 >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 0x7f53800d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:23.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.781+0000 7f537ffff640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 0x7f5380162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.782+0000 7f537effd640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53800b4e40 0x7f5380161f00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f53800b69f0 tx=0x7f537402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a69902b874c4898 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.782+0000 7f537e7fc640 1 -- 172.21.15.1:0/4137226731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374031640 con 0x7f53800b4e40 2024-03-24T10:36:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.782+0000 7f537e7fc640 1 -- 172.21.15.1:0/4137226731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53740317a0 con 0x7f53800b4e40 2024-03-24T10:36:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.782+0000 7f537e7fc640 1 -- 172.21.15.1:0/4137226731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374031b30 con 0x7f53800b4e40 2024-03-24T10:36:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 -- 172.21.15.1:0/4137226731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53800b4e40 msgr2=0x7f5380161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53800b4e40 0x7f5380161f00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f53800b69f0 tx=0x7f537402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 -- 172.21.15.1:0/4137226731 shutdown_connections 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801619d0 0x7f5380162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 --2- 172.21.15.1:0/4137226731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53800b4e40 0x7f5380161f00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 --2- 172.21.15.1:0/4137226731 >> v2:172.21.15.3:3300/0 conn(0x7f53801615f0 0x7f53800d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 -- 172.21.15.1:0/4137226731 >> 172.21.15.1:0/4137226731 conn(0x7f53800d3ef0 msgr2=0x7f53800d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 -- 172.21.15.1:0/4137226731 shutdown_connections 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.783+0000 7f5385c88640 1 -- 172.21.15.1:0/4137226731 wait complete. 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.784+0000 7f5385c88640 1 Processor -- start 2024-03-24T10:36:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.784+0000 7f5385c88640 1 -- start start 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801615f0 0x7f53801555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53801619d0 0x7f53801f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f53800b6ae0 con 0x7f53801560a0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f53800b8080 con 0x7f53801619d0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f5385c88640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f53801f4e50 con 0x7f53801615f0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58458/0 (socket says 172.21.15.1:58458) 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 -- 172.21.15.1:0/2252147226 learned_addr learned my addr 172.21.15.1:0/2252147226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537effd640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53801619d0 0x7f53801f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537f7fe640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801615f0 0x7f53801555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 -- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801615f0 msgr2=0x7f53801555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801615f0 0x7f53801555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 -- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53801619d0 msgr2=0x7f53801f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53801619d0 0x7f53801f4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.785+0000 7f537ffff640 1 -- 172.21.15.1:0/2252147226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53740094d0 con 0x7f53801560a0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f537ffff640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f537800cd50 tx=0x7f537800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537800f680 con 0x7f53801560a0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537800f800 con 0x7f53801560a0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5380157e70 con 0x7f53801560a0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537800fbb0 con 0x7f53801560a0 2024-03-24T10:36:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.786+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53801568b0 con 0x7f53801560a0 2024-03-24T10:36:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.787+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f538004f470 con 0x7f53801560a0 2024-03-24T10:36:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.787+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f537800b040 con 0x7f53801560a0 2024-03-24T10:36:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.788+0000 7f5367fff640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 0x7f536c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.788+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f537808aba0 con 0x7f53801560a0 2024-03-24T10:36:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.788+0000 7f537f7fe640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 0x7f536c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.788+0000 7f537f7fe640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 0x7f536c06dd00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f53800b9d10 tx=0x7f5370008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:23.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.792+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f53780555b0 con 0x7f53801560a0 2024-03-24T10:36:23.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.975+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f538015c960 con 0x7f53801560a0 2024-03-24T10:36:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.976+0000 7f5367fff640 1 -- 172.21.15.1:0/2252147226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7f5378058c70 con 0x7f53801560a0 2024-03-24T10:36:23.976 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 msgr2=0x7f536c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 0x7f536c06dd00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f53800b9d10 tx=0x7f5370008040 comp rx=0 tx=0).stop 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 msgr2=0x7f5380157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f537800cd50 tx=0x7f537800a650 comp rx=0 tx=0).stop 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 shutdown_connections 2024-03-24T10:36:23.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f536c06b840 0x7f536c06dd00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53801560a0 0x7f5380157930 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53801619d0 0x7f53801f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 --2- 172.21.15.1:0/2252147226 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53801615f0 0x7f53801555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 >> 172.21.15.1:0/2252147226 conn(0x7f53800d3ef0 msgr2=0x7f53800d24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 shutdown_connections 2024-03-24T10:36:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:23.978+0000 7f5385c88640 1 -- 172.21.15.1:0/2252147226 wait complete. 2024-03-24T10:36:23.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:36:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.093+0000 7f6062088640 1 Processor -- start 2024-03-24T10:36:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.093+0000 7f6062088640 1 -- start start 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f6062088640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c0d2a20 0x7f605c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f6062088640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f6062088640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c0b0b70 con 0x7f605c15e390 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f6062088640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f605c0d1d60 con 0x7f605c0d2a20 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f6062088640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f605c0b0de0 con 0x7f605c0b2310 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c0d2a20 0x7f605c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c0d2a20 0x7f605c0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58468/0 (socket says 172.21.15.1:58468) 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 -- 172.21.15.1:0/1540138450 learned_addr learned my addr 172.21.15.1:0/1540138450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605bfff640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 -- 172.21.15.1:0/1540138450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 msgr2=0x7f605c0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c0cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 -- 172.21.15.1:0/1540138450 >> v1:172.21.15.3:6789/0 conn(0x7f605c15e390 legacy=0x7f605c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605affd640 1 -- 172.21.15.1:0/1540138450 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c15e770 con 0x7f605c0d2a20 2024-03-24T10:36:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.094+0000 7f605bfff640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.095+0000 7f605affd640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0cead0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f605002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b21f7d979a7b7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.095+0000 7f605a7fc640 1 -- 172.21.15.1:0/1540138450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050030840 con 0x7f605c0d2a20 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.095+0000 7f605a7fc640 1 -- 172.21.15.1:0/1540138450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60500309a0 con 0x7f605c0d2a20 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.095+0000 7f605a7fc640 1 -- 172.21.15.1:0/1540138450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050030d30 con 0x7f605c0d2a20 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 -- 172.21.15.1:0/1540138450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 msgr2=0x7f605c0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0cead0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6050009870 tx=0x7f605002b980 comp rx=0 tx=0).stop 2024-03-24T10:36:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 -- 172.21.15.1:0/1540138450 shutdown_connections 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 --2- 172.21.15.1:0/1540138450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0cead0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 -- 172.21.15.1:0/1540138450 >> 172.21.15.1:0/1540138450 conn(0x7f605c0d3ee0 msgr2=0x7f605c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 -- 172.21.15.1:0/1540138450 shutdown_connections 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.096+0000 7f6062088640 1 -- 172.21.15.1:0/1540138450 wait complete. 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.097+0000 7f6062088640 1 Processor -- start 2024-03-24T10:36:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.097+0000 7f6062088640 1 -- start start 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f605c15e390 0x7f605c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0ce5e0 con 0x7f605c0d2a20 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f605c1548c0 con 0x7f605c15e390 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f6062088640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f605c0ce300 con 0x7f605c0b2310 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58474/0 (socket says 172.21.15.1:58474) 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 -- 172.21.15.1:0/437178579 learned_addr learned my addr 172.21.15.1:0/437178579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605bfff640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f605c15e390 0x7f605c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605b7fe640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 -- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 msgr2=0x7f605c15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 -- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f605c15e390 msgr2=0x7f605c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f605c15e390 0x7f605c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 -- 172.21.15.1:0/437178579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60500094d0 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605b7fe640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.098+0000 7f605affd640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6050000c00 tx=0x7f6050004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.099+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050039c90 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.099+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6050039df0 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.099+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c0b4480 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.099+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050043640 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.099+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c0b4a40 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.100+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c0b42e0 con 0x7f605c0d2a20 2024-03-24T10:36:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.100+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f60500437a0 con 0x7f605c0d2a20 2024-03-24T10:36:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.101+0000 7f6043fff640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 0x7f604806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.101+0000 7f605b7fe640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 0x7f604806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.101+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f605003e070 con 0x7f605c0d2a20 2024-03-24T10:36:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.101+0000 7f605b7fe640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 0x7f604806dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f604c001660 tx=0x7f604c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.105+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f605c0b42e0 con 0x7f605c0d2a20 2024-03-24T10:36:24.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.346+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f605c0b42e0 con 0x7f605c0d2a20 2024-03-24T10:36:24.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.346+0000 7f6043fff640 1 -- 172.21.15.1:0/437178579 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7f605c0b42e0 con 0x7f605c0d2a20 2024-03-24T10:36:24.347 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-24T10:36:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 msgr2=0x7f604806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 0x7f604806dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f604c001660 tx=0x7f604c009040 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 msgr2=0x7f605c0c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6050000c00 tx=0x7f6050004730 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 shutdown_connections 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f604806b810 0x7f604806dcd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f605c15e390 0x7f605c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0d2a20 0x7f605c0c80a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 --2- 172.21.15.1:0/437178579 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f605c0b2310 0x7f605c15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 >> 172.21.15.1:0/437178579 conn(0x7f605c0d3ee0 msgr2=0x7f605c0cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 shutdown_connections 2024-03-24T10:36:24.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.348+0000 7f6062088640 1 -- 172.21.15.1:0/437178579 wait complete. 2024-03-24T10:36:24.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:24.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.463+0000 7f295c325640 1 Processor -- start 2024-03-24T10:36:24.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.463+0000 7f295c325640 1 -- start start 2024-03-24T10:36:24.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295c325640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f29540d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295c325640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f29540b4ba0 0x7f29540d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295c325640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295415e770 con 0x7f29540d1000 2024-03-24T10:36:24.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295c325640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f29540b3190 con 0x7f29540b4ba0 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295c325640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f29540d4e40 con 0x7f295415e390 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f29598c0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f29540d1000 0x7f29540cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54928/0 (socket says 172.21.15.1:54928) 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f29598c0640 1 -- 172.21.15.1:0/2051617733 learned_addr learned my addr 172.21.15.1:0/2051617733 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295a8c2640 1 --2- 172.21.15.1:0/2051617733 >> v2:172.21.15.3:3300/0 conn(0x7f29540b4ba0 0x7f29540d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.464+0000 7f295a0c1640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f29540d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978781086 0 0) 0x7f2944001110 con 0x7f29540d1000 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2940003520 con 0x7f29540d1000 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3322044582 0 0) 0x7f2940003520 con 0x7f29540d1000 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 msgr2=0x7f29540d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f29540d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 >> v2:172.21.15.3:3300/0 conn(0x7f29540b4ba0 msgr2=0x7f29540d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 --2- 172.21.15.1:0/2051617733 >> v2:172.21.15.3:3300/0 conn(0x7f29540b4ba0 0x7f29540d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29540d1e40 con 0x7f29540d1000 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f2944001800 con 0x7f29540d1000 2024-03-24T10:36:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f295a8c2640 1 --2- 172.21.15.1:0/2051617733 >> v2:172.21.15.3:3300/0 conn(0x7f29540b4ba0 0x7f29540d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 >> v1:172.21.15.1:6789/0 conn(0x7f29540d1000 legacy=0x7f29540cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2940003970 0x7f2940003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f2940004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f2940005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29540d4e40 con 0x7f29400047c0 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f294000b980 con 0x7f2940003970 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f294000bb40 con 0x7f295415e390 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f2940004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f295a0c1640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2940003970 0x7f2940003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f295a8c2640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f2940005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 -- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 msgr2=0x7f2940005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f295415e390 0x7f2940005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 -- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2940003970 msgr2=0x7f2940003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2940003970 0x7f2940003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.465+0000 7f29598c0640 1 -- 172.21.15.1:0/2051617733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940003520 con 0x7f29400047c0 2024-03-24T10:36:24.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.466+0000 7f29598c0640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f2940004c40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f29540d1440 tx=0x7f29400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855f9d707b36685f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.466+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944005ad0 con 0x7f29400047c0 2024-03-24T10:36:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.466+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2944005c30 con 0x7f29400047c0 2024-03-24T10:36:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.466+0000 7f29590bf640 1 -- 172.21.15.1:0/2051617733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294402c890 con 0x7f29400047c0 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 -- 172.21.15.1:0/2051617733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 msgr2=0x7f2940004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 --2- 172.21.15.1:0/2051617733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f2940004c40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f29540d1440 tx=0x7f29400057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295a8c2640 1 -- 172.21.15.1:0/2051617733 reap_dead start 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 -- 172.21.15.1:0/2051617733 shutdown_connections 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 -- 172.21.15.1:0/2051617733 >> 172.21.15.1:0/2051617733 conn(0x7f29540d3ec0 msgr2=0x7f29540d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 -- 172.21.15.1:0/2051617733 shutdown_connections 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.467+0000 7f295c325640 1 -- 172.21.15.1:0/2051617733 wait complete. 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.468+0000 7f295c325640 1 Processor -- start 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.468+0000 7f295c325640 1 -- start start 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2954151440 0x7f29540cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29541518c0 0x7f29540cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29540cbc40 con 0x7f29400047c0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f29540cb030 con 0x7f2954151440 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295c325640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f29540cb310 con 0x7f29541518c0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58488/0 (socket says 172.21.15.1:58488) 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 -- 172.21.15.1:0/1879631017 learned_addr learned my addr 172.21.15.1:0/1879631017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f29598c0640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2954151440 0x7f29540cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a8c2640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29541518c0 0x7f29540cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 -- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29541518c0 msgr2=0x7f29540cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29541518c0 0x7f29540cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 -- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2954151440 msgr2=0x7f29540cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2954151440 0x7f29540cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 -- 172.21.15.1:0/1879631017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29540cb5f0 con 0x7f29400047c0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.469+0000 7f295a0c1640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f29500118e0 tx=0x7f2950011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.470+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295000faf0 con 0x7f29400047c0 2024-03-24T10:36:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.470+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f295000fc50 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.470+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29540cb8d0 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.470+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950014440 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.470+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2954160440 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.471+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29540cb450 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.471+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f29500145a0 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.472+0000 7f294affd640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 0x7f293806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.472+0000 7f29598c0640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 0x7f293806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.472+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f295008aa30 con 0x7f29400047c0 2024-03-24T10:36:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.472+0000 7f29598c0640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 0x7f293806dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2944034000 tx=0x7f2944007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.476+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29540cb450 con 0x7f29400047c0 2024-03-24T10:36:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.695+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29540cb450 con 0x7f29400047c0 2024-03-24T10:36:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.695+0000 7f294affd640 1 -- 172.21.15.1:0/1879631017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f29540cb450 con 0x7f29400047c0 2024-03-24T10:36:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:24.696 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:36:24.696 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:36:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 msgr2=0x7f293806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 0x7f293806dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2944034000 tx=0x7f2944007000 comp rx=0 tx=0).stop 2024-03-24T10:36:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 msgr2=0x7f295415f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f29500118e0 tx=0x7f2950011da0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 shutdown_connections 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f293806b810 0x7f293806dcd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f29541518c0 0x7f29540cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2954151440 0x7f29540cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 --2- 172.21.15.1:0/1879631017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29400047c0 0x7f295415f260 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.697+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 >> 172.21.15.1:0/1879631017 conn(0x7f29540d3ec0 msgr2=0x7f29540d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.698+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 shutdown_connections 2024-03-24T10:36:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.698+0000 7f295c325640 1 -- 172.21.15.1:0/1879631017 wait complete. 2024-03-24T10:36:24.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:24.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.810+0000 7f39ebfa0640 1 Processor -- start 2024-03-24T10:36:24.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.810+0000 7f39ebfa0640 1 -- start start 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39ebfa0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39e415e390 0x7f39e40d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39ebfa0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39e415e770 con 0x7f39e40d1000 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39ebfa0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39e40d4e40 con 0x7f39e40b4ba0 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39ebfa0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f39e40b3190 con 0x7f39e415e390 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39e415e390 0x7f39e40d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39e415e390 0x7f39e40d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58492/0 (socket says 172.21.15.1:58492) 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e953b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f39e40d1000 0x7f39e40cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54942/0 (socket says 172.21.15.1:54942) 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 -- 172.21.15.1:0/2159248708 learned_addr learned my addr 172.21.15.1:0/2159248708 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 -- 172.21.15.1:0/2159248708 >> v1:172.21.15.3:6789/0 conn(0x7f39e40b4ba0 legacy=0x7f39e40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 -- 172.21.15.1:0/2159248708 >> v1:172.21.15.1:6789/0 conn(0x7f39e40d1000 legacy=0x7f39e40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e8d3a640 1 -- 172.21.15.1:0/2159248708 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187950143 0 0) 0x7f39d4001110 con 0x7f39e40d1000 2024-03-24T10:36:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.811+0000 7f39e9d3c640 1 -- 172.21.15.1:0/2159248708 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e40d1e40 con 0x7f39e415e390 2024-03-24T10:36:24.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.812+0000 7f39e9d3c640 1 --2- 172.21.15.1:0/2159248708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e415e390 0x7f39e40d24a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009870 tx=0x7f39e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2035fe6b70ccf044 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.812+0000 7f39e8d3a640 1 -- 172.21.15.1:0/2159248708 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e00317e0 con 0x7f39e415e390 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.812+0000 7f39e8d3a640 1 -- 172.21.15.1:0/2159248708 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e0031940 con 0x7f39e415e390 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.812+0000 7f39e8d3a640 1 -- 172.21.15.1:0/2159248708 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0031cd0 con 0x7f39e415e390 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 -- 172.21.15.1:0/2159248708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e415e390 msgr2=0x7f39e40d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/2159248708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e415e390 0x7f39e40d24a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009870 tx=0x7f39e002b880 comp rx=0 tx=0).stop 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 -- 172.21.15.1:0/2159248708 shutdown_connections 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/2159248708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e415e390 0x7f39e40d24a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 -- 172.21.15.1:0/2159248708 >> 172.21.15.1:0/2159248708 conn(0x7f39e40d3ec0 msgr2=0x7f39e40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 -- 172.21.15.1:0/2159248708 shutdown_connections 2024-03-24T10:36:24.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.813+0000 7f39ebfa0640 1 -- 172.21.15.1:0/2159248708 wait complete. 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.814+0000 7f39ebfa0640 1 Processor -- start 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.814+0000 7f39ebfa0640 1 -- start start 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f39e40b4ba0 0x7f39e4152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 0x7f39e40cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f39e4152880 con 0x7f39e40d1000 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f39e415a540 con 0x7f39e415e390 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ebfa0640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f39e415a820 con 0x7f39e40b4ba0 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58494/0 (socket says 172.21.15.1:58494) 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 -- 172.21.15.1:0/898318104 learned_addr learned my addr 172.21.15.1:0/898318104 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 -- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f39e40b4ba0 msgr2=0x7f39e4152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f39e40b4ba0 0x7f39e4152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ea53d640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 0x7f39e40cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 -- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 msgr2=0x7f39e40cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 0x7f39e40cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 -- 172.21.15.1:0/898318104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e00094d0 con 0x7f39e40d1000 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39ea53d640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 0x7f39e40cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.815+0000 7f39e953b640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c410 tx=0x7f39d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.816+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40074e0 con 0x7f39e40d1000 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.816+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d4007660 con 0x7f39e40d1000 2024-03-24T10:36:24.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.816+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e415ab00 con 0x7f39e40d1000 2024-03-24T10:36:24.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.816+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d4004a20 con 0x7f39e40d1000 2024-03-24T10:36:24.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.816+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e415b0c0 con 0x7f39e40d1000 2024-03-24T10:36:24.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.817+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e415b2b0 con 0x7f39e40d1000 2024-03-24T10:36:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.817+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f39d4004b80 con 0x7f39e40d1000 2024-03-24T10:36:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.818+0000 7f39da7fc640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 0x7f39c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.818+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f39d4089f50 con 0x7f39e40d1000 2024-03-24T10:36:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.818+0000 7f39e9d3c640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 0x7f39c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:24.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.818+0000 7f39e9d3c640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 0x7f39c806dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f39e0036000 tx=0x7f39e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:24.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:24.822+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f39d4058410 con 0x7f39e40d1000 2024-03-24T10:36:25.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.158+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39e415e770 con 0x7f39e40d1000 2024-03-24T10:36:25.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.159+0000 7f39da7fc640 1 -- 172.21.15.1:0/898318104 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f39e415e770 con 0x7f39e40d1000 2024-03-24T10:36:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:25.160 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 msgr2=0x7f39c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 0x7f39c806dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f39e0036000 tx=0x7f39e0030000 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 msgr2=0x7f39e41f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c410 tx=0x7f39d400c8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 shutdown_connections 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f39c806b810 0x7f39c806dcd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e415e390 0x7f39e40cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39e40d1000 0x7f39e41f3bf0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 --2- 172.21.15.1:0/898318104 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f39e40b4ba0 0x7f39e4152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 >> 172.21.15.1:0/898318104 conn(0x7f39e40d3ec0 msgr2=0x7f39e40b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:25.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 shutdown_connections 2024-03-24T10:36:25.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.161+0000 7f39ebfa0640 1 -- 172.21.15.1:0/898318104 wait complete. 2024-03-24T10:36:25.173 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T10:36:25.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.280+0000 7feddcb81640 1 Processor -- start 2024-03-24T10:36:25.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.280+0000 7feddcb81640 1 -- start start 2024-03-24T10:36:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedd815e390 0x7fedd80d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fedd80d1000 0x7fedd80d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd80b3320 0x7fedd80cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fedd80b3190 con 0x7fedd815e390 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fedd815e770 con 0x7fedd80d1000 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7feddcb81640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fedd80d4e40 con 0x7fedd80b3320 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.281+0000 7fedd659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedd815e390 0x7fedd80d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedd815e390 0x7fedd80d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58502/0 (socket says 172.21.15.1:58502) 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 -- 172.21.15.1:0/47908320 learned_addr learned my addr 172.21.15.1:0/47908320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd5d9b640 1 --2- 172.21.15.1:0/47908320 >> v2:172.21.15.3:3300/0 conn(0x7fedd80d1000 0x7fedd80d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd6d9d640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd80b3320 0x7fedd80cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 -- 172.21.15.1:0/47908320 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd80b3320 msgr2=0x7fedd80cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd80b3320 0x7fedd80cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 -- 172.21.15.1:0/47908320 >> v2:172.21.15.3:3300/0 conn(0x7fedd80d1000 msgr2=0x7fedd80d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 --2- 172.21.15.1:0/47908320 >> v2:172.21.15.3:3300/0 conn(0x7fedd80d1000 0x7fedd80d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.282+0000 7fedd659c640 1 -- 172.21.15.1:0/47908320 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd80d1e40 con 0x7fedd815e390 2024-03-24T10:36:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.283+0000 7fedd659c640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd815e390 0x7fedd80d24a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fedd806b4e0 tx=0x7fedc802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb45c62af696aa6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.283+0000 7fedd559a640 1 -- 172.21.15.1:0/47908320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8031750 con 0x7fedd815e390 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.283+0000 7fedd559a640 1 -- 172.21.15.1:0/47908320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc80318b0 con 0x7fedd815e390 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.283+0000 7fedd559a640 1 -- 172.21.15.1:0/47908320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8031c40 con 0x7fedd815e390 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 -- 172.21.15.1:0/47908320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd815e390 msgr2=0x7fedd80d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd815e390 0x7fedd80d24a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fedd806b4e0 tx=0x7fedc802b7f0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 -- 172.21.15.1:0/47908320 shutdown_connections 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd80b3320 0x7fedd80cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 --2- 172.21.15.1:0/47908320 >> v2:172.21.15.3:3300/0 conn(0x7fedd80d1000 0x7fedd80d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 --2- 172.21.15.1:0/47908320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd815e390 0x7fedd80d24a0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 -- 172.21.15.1:0/47908320 >> 172.21.15.1:0/47908320 conn(0x7fedd80d3ec0 msgr2=0x7fedd80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 -- 172.21.15.1:0/47908320 shutdown_connections 2024-03-24T10:36:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.284+0000 7feddcb81640 1 -- 172.21.15.1:0/47908320 wait complete. 2024-03-24T10:36:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.286+0000 7feddcb81640 1 Processor -- start 2024-03-24T10:36:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.286+0000 7feddcb81640 1 -- start start 2024-03-24T10:36:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedd80d1000 0x7fedd8155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd815e390 0x7fedd815be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80cbee0 con 0x7fedd80b3320 2024-03-24T10:36:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80cd5f0 con 0x7fedd80d1000 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7feddcb81640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fedd80cc550 con 0x7fedd815e390 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58516/0 (socket says 172.21.15.1:58516) 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 -- 172.21.15.1:0/793461052 learned_addr learned my addr 172.21.15.1:0/793461052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd5d9b640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedd80d1000 0x7fedd8155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 -- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd815e390 msgr2=0x7fedd815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd815e390 0x7fedd815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 -- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedd80d1000 msgr2=0x7fedd8155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedd80d1000 0x7fedd8155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.287+0000 7fedd659c640 1 -- 172.21.15.1:0/793461052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc80094d0 con 0x7fedd80b3320 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7fedd659c640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fedc802b830 tx=0x7fedc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8033850 con 0x7fedd80b3320 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc80339b0 con 0x7fedd80b3320 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd80cc830 con 0x7fedd80b3320 2024-03-24T10:36:25.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8033d60 con 0x7fedd80b3320 2024-03-24T10:36:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.288+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd80ccdf0 con 0x7fedd80b3320 2024-03-24T10:36:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.289+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd804f4e0 con 0x7fedd80b3320 2024-03-24T10:36:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.290+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fedc803a040 con 0x7fedd80b3320 2024-03-24T10:36:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.290+0000 7fedc6ffd640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 0x7fedb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.290+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fedc80abae0 con 0x7fedd80b3320 2024-03-24T10:36:25.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.290+0000 7fedd5d9b640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 0x7fedb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.291+0000 7fedd5d9b640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 0x7fedb806dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fedd80cd800 tx=0x7fedcc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.295+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fedc8076590 con 0x7fedd80b3320 2024-03-24T10:36:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.513+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedd80d2a40 con 0x7fedd80b3320 2024-03-24T10:36:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.514+0000 7fedc6ffd640 1 -- 172.21.15.1:0/793461052 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7fedc804d090 con 0x7fedd80b3320 2024-03-24T10:36:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:25.514 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:36:25.514 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 msgr2=0x7fedb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 0x7fedb806dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fedd80cd800 tx=0x7fedcc008040 comp rx=0 tx=0).stop 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 msgr2=0x7fedd80cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fedc802b830 tx=0x7fedc80046c0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 shutdown_connections 2024-03-24T10:36:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fedb806b810 0x7fedb806dcd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fedd815e390 0x7fedd815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedd80d1000 0x7fedd8155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 --2- 172.21.15.1:0/793461052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd80b3320 0x7fedd80cb9a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 >> 172.21.15.1:0/793461052 conn(0x7fedd80d3ec0 msgr2=0x7fedd80b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 shutdown_connections 2024-03-24T10:36:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.516+0000 7feddcb81640 1 -- 172.21.15.1:0/793461052 wait complete. 2024-03-24T10:36:25.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.629+0000 7f6663320640 1 Processor -- start 2024-03-24T10:36:25.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 -- start start 2024-03-24T10:36:25.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f665c0adc70 0x7f665c0ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f665c0b3d00 0x7f665c0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c1553c0 con 0x7f665c154fe0 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f665c06d250 con 0x7f665c0b3d00 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f6663320640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f665c06c7e0 con 0x7f665c0adc70 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.630+0000 7f66618bd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f665c0b3d00 0x7f665c0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f665c0b3d00 0x7f665c0b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58526/0 (socket says 172.21.15.1:58526) 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66610bc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f665c154fe0 0x7f665c06b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54954/0 (socket says 172.21.15.1:54954) 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 -- 172.21.15.1:0/1238158512 learned_addr learned my addr 172.21.15.1:0/1238158512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66608bb640 1 --2- 172.21.15.1:0/1238158512 >> v2:172.21.15.3:3300/0 conn(0x7f665c0adc70 0x7f665c0ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 -- 172.21.15.1:0/1238158512 >> v2:172.21.15.3:3300/0 conn(0x7f665c0adc70 msgr2=0x7f665c0ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 --2- 172.21.15.1:0/1238158512 >> v2:172.21.15.3:3300/0 conn(0x7f665c0adc70 0x7f665c0ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 -- 172.21.15.1:0/1238158512 >> v1:172.21.15.1:6789/0 conn(0x7f665c154fe0 legacy=0x7f665c06b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f66618bd640 1 -- 172.21.15.1:0/1238158512 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c0ae5d0 con 0x7f665c0b3d00 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.631+0000 7f6653fff640 1 -- 172.21.15.1:0/1238158512 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452621600 0 0) 0x7f665c1553c0 con 0x7f665c154fe0 2024-03-24T10:36:25.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f66618bd640 1 --2- 172.21.15.1:0/1238158512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c0b4170 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f665c0b3a20 tx=0x7f665402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46198cfc56890358 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6653fff640 1 -- 172.21.15.1:0/1238158512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665402cb20 con 0x7f665c0b3d00 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6653fff640 1 -- 172.21.15.1:0/1238158512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f665402cc80 con 0x7f665c0b3d00 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6653fff640 1 -- 172.21.15.1:0/1238158512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540343f0 con 0x7f665c0b3d00 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 -- 172.21.15.1:0/1238158512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 msgr2=0x7f665c0b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 --2- 172.21.15.1:0/1238158512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c0b4170 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f665c0b3a20 tx=0x7f665402bf80 comp rx=0 tx=0).stop 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 -- 172.21.15.1:0/1238158512 shutdown_connections 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 --2- 172.21.15.1:0/1238158512 >> v2:172.21.15.3:3300/0 conn(0x7f665c0adc70 0x7f665c0ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 --2- 172.21.15.1:0/1238158512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c0b4170 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 -- 172.21.15.1:0/1238158512 >> 172.21.15.1:0/1238158512 conn(0x7f665c0d0570 msgr2=0x7f665c0d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 -- 172.21.15.1:0/1238158512 shutdown_connections 2024-03-24T10:36:25.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.632+0000 7f6663320640 1 -- 172.21.15.1:0/1238158512 wait complete. 2024-03-24T10:36:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.634+0000 7f6663320640 1 Processor -- start 2024-03-24T10:36:25.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.634+0000 7f6663320640 1 -- start start 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f665c0adc70 0x7f665c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f665c154fe0 0x7f665c143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f665c0cee40 con 0x7f665c0b3d00 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f665c155a50 con 0x7f665c154fe0 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f6663320640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f665c144410 con 0x7f665c0adc70 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f66608bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f66608bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58532/0 (socket says 172.21.15.1:58532) 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f66608bb640 1 -- 172.21.15.1:0/3934683309 learned_addr learned my addr 172.21.15.1:0/3934683309 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f66610bc640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f665c0adc70 0x7f665c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.635+0000 7f66618bd640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f665c154fe0 0x7f665c143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 -- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f665c0adc70 msgr2=0x7f665c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f665c0adc70 0x7f665c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 -- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f665c154fe0 msgr2=0x7f665c143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f665c154fe0 0x7f665c143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 -- 172.21.15.1:0/3934683309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6654009b30 con 0x7f665c0b3d00 2024-03-24T10:36:25.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f66608bb640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f665800d830 tx=0x7f665800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665800b660 con 0x7f665c0b3d00 2024-03-24T10:36:25.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f665800b7c0 con 0x7f665c0b3d00 2024-03-24T10:36:25.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f665c0ce4a0 con 0x7f665c0b3d00 2024-03-24T10:36:25.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665800bb50 con 0x7f665c0b3d00 2024-03-24T10:36:25.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.636+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f665c0cea60 con 0x7f665c0b3d00 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.637+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f665801e460 con 0x7f665c0b3d00 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.637+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665c04f4e0 con 0x7f665c0b3d00 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.638+0000 7f6651ffb640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 0x7f664006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.638+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f665808a710 con 0x7f665c0b3d00 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.638+0000 7f66610bc640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 0x7f664006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:25.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.638+0000 7f66610bc640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 0x7f664006dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:25.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:25.642+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6658055270 con 0x7f665c0b3d00 2024-03-24T10:36:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.068+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f665c0b3a50 con 0x7f665c0b3d00 2024-03-24T10:36:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.069+0000 7f6651ffb640 1 -- 172.21.15.1:0/3934683309 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f6658058930 con 0x7f665c0b3d00 2024-03-24T10:36:26.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:26.069 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:35:50.983041+0000","modified":"2024-03-24T10:35:52.981728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1119214087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":1119214087},{"type":"v1","addr":"172.21.15.1:6837","nonce":1119214087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-03-24T10:36:26.069 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 msgr2=0x7f664006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 0x7f664006dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c007de0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 msgr2=0x7f665c155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f665800d830 tx=0x7f665800dcf0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 shutdown_connections 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f664006b810 0x7f664006dcd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f665c154fe0 0x7f665c143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.071+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f665c0b3d00 0x7f665c155510 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.072+0000 7f6663320640 1 --2- 172.21.15.1:0/3934683309 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f665c0adc70 0x7f665c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.072+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 >> 172.21.15.1:0/3934683309 conn(0x7f665c0d0570 msgr2=0x7f665c0b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.072+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 shutdown_connections 2024-03-24T10:36:26.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.072+0000 7f6663320640 1 -- 172.21.15.1:0/3934683309 wait complete. 2024-03-24T10:36:26.087 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:26.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:36:26.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:36:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.206+0000 7f03bb655640 1 Processor -- start 2024-03-24T10:36:26.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03bb655640 1 -- start start 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03bb655640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03b415b180 0x7f03b40b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03bb655640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b4067530 con 0x7f03b406bc20 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03bb655640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b40b1780 con 0x7f03b40b12c0 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03bb655640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f03b406b140 con 0x7f03b415b180 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03b93f1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03b415b180 0x7f03b40b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03b9bf2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03b406bc20 0x7f03b406c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54966/0 (socket says 172.21.15.1:54966) 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03b93f1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03b415b180 0x7f03b40b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58542/0 (socket says 172.21.15.1:58542) 2024-03-24T10:36:26.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.207+0000 7f03b9bf2640 1 -- 172.21.15.1:0/2364211807 learned_addr learned my addr 172.21.15.1:0/2364211807 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.208+0000 7f03b93f1640 1 -- 172.21.15.1:0/2364211807 >> v1:172.21.15.3:6789/0 conn(0x7f03b40b12c0 legacy=0x7f03b40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.208+0000 7f03b93f1640 1 -- 172.21.15.1:0/2364211807 >> v1:172.21.15.1:6789/0 conn(0x7f03b406bc20 legacy=0x7f03b406c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.208+0000 7f03b3fff640 1 -- 172.21.15.1:0/2364211807 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344171681 0 0) 0x7f03ac001110 con 0x7f03b406bc20 2024-03-24T10:36:26.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.208+0000 7f03b93f1640 1 -- 172.21.15.1:0/2364211807 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b406b420 con 0x7f03b415b180 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.208+0000 7f03b93f1640 1 --2- 172.21.15.1:0/2364211807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b415b180 0x7f03b40b4ff0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3af6e667637c6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03b3fff640 1 -- 172.21.15.1:0/2364211807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a402cc60 con 0x7f03b415b180 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03b3fff640 1 -- 172.21.15.1:0/2364211807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a402cdc0 con 0x7f03b415b180 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03b3fff640 1 -- 172.21.15.1:0/2364211807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4034660 con 0x7f03b415b180 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 -- 172.21.15.1:0/2364211807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b415b180 msgr2=0x7f03b40b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 --2- 172.21.15.1:0/2364211807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b415b180 0x7f03b40b4ff0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009d90 tx=0x7f03a402beb0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 -- 172.21.15.1:0/2364211807 shutdown_connections 2024-03-24T10:36:26.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 --2- 172.21.15.1:0/2364211807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b415b180 0x7f03b40b4ff0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 -- 172.21.15.1:0/2364211807 >> 172.21.15.1:0/2364211807 conn(0x7f03b40d0320 msgr2=0x7f03b40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 -- 172.21.15.1:0/2364211807 shutdown_connections 2024-03-24T10:36:26.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.209+0000 7f03bb655640 1 -- 172.21.15.1:0/2364211807 wait complete. 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.210+0000 7f03bb655640 1 Processor -- start 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.210+0000 7f03bb655640 1 -- start start 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03b406bc20 0x7f03b40cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 0x7f03b40c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f03b40c9f50 con 0x7f03b40b12c0 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f03b40c7af0 con 0x7f03b415b180 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03bb655640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f03b40ca720 con 0x7f03b406bc20 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03b8bf0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03b8bf0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58558/0 (socket says 172.21.15.1:58558) 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.211+0000 7f03b8bf0640 1 -- 172.21.15.1:0/1405154533 learned_addr learned my addr 172.21.15.1:0/1405154533 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b93f1640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03b406bc20 0x7f03b40cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 -- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03b406bc20 msgr2=0x7f03b40cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b9bf2640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 0x7f03b40c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03b406bc20 0x7f03b40cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 -- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 msgr2=0x7f03b40c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 0x7f03b40c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 -- 172.21.15.1:0/1405154533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a4009a70 con 0x7f03b40b12c0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b9bf2640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 0x7f03b40c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b8bf0640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c410 tx=0x7f03a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a80035f0 con 0x7f03b40b12c0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a8003750 con 0x7f03b40b12c0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8003b40 con 0x7f03b40b12c0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b40ca160 con 0x7f03b40b12c0 2024-03-24T10:36:26.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.212+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b40cd190 con 0x7f03b40b12c0 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.213+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b404f4e0 con 0x7f03b40b12c0 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.213+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03a8003ca0 con 0x7f03b40b12c0 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.214+0000 7f03b1ffb640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 0x7f03a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.214+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7f03a80898a0 con 0x7f03b40b12c0 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.214+0000 7f03b93f1640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 0x7f03a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.215+0000 7f03b93f1640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 0x7f03a006dcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f03a4036000 tx=0x7f03a4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.218+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03a80542c0 con 0x7f03b40b12c0 2024-03-24T10:36:26.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.402+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03b4066f40 con 0x7f03b40b12c0 2024-03-24T10:36:26.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.403+0000 7f03b1ffb640 1 -- 172.21.15.1:0/1405154533 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16537 (secure 0 0 0) 0x7f03a8017020 con 0x7f03b40b12c0 2024-03-24T10:36:26.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:26.404 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":50,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:35:52.625258+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:35:49.256489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T10:35:49.948166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 msgr2=0x7f03a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 0x7f03a006dcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f03a4036000 tx=0x7f03a4030000 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 msgr2=0x7f03b40c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c410 tx=0x7f03a800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 shutdown_connections 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f03a006b810 0x7f03a006dcd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03b415b180 0x7f03b40c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03b40b12c0 0x7f03b40c9a10 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 --2- 172.21.15.1:0/1405154533 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f03b406bc20 0x7f03b40cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 >> 172.21.15.1:0/1405154533 conn(0x7f03b40d0320 msgr2=0x7f03b406d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 shutdown_connections 2024-03-24T10:36:26.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.405+0000 7f03bb655640 1 -- 172.21.15.1:0/1405154533 wait complete. 2024-03-24T10:36:26.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T10:36:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.560+0000 7fbcc46da640 1 Processor -- start 2024-03-24T10:36:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.560+0000 7fbcc46da640 1 -- start start 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc46da640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcbc15e390 0x7fbcbc0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc46da640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc0d1000 0x7fbcbc0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc46da640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcbc15e770 con 0x7fbcbc0b3320 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc46da640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbcbc0b3190 con 0x7fbcbc15e390 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc46da640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc0d4e40 con 0x7fbcbc0d1000 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc2476640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcbc15e390 0x7fbcbc0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.561+0000 7fbcc2476640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcbc15e390 0x7fbcbc0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58560/0 (socket says 172.21.15.1:58560) 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc2476640 1 -- 172.21.15.1:0/179517803 learned_addr learned my addr 172.21.15.1:0/179517803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc1c75640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc0d1000 0x7fbcbc0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc2476640 1 -- 172.21.15.1:0/179517803 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc0d1000 msgr2=0x7fbcbc0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc2476640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc0d1000 0x7fbcbc0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc2476640 1 -- 172.21.15.1:0/179517803 >> v1:172.21.15.3:6789/0 conn(0x7fbcbc0b3320 legacy=0x7fbcbc0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.562+0000 7fbcc2476640 1 -- 172.21.15.1:0/179517803 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc0d1e40 con 0x7fbcbc15e390 2024-03-24T10:36:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc2476640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc15e390 0x7fbcbc0d24a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0b1beab47fc2c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc1474640 1 -- 172.21.15.1:0/179517803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac031740 con 0x7fbcbc15e390 2024-03-24T10:36:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc1474640 1 -- 172.21.15.1:0/179517803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcac0318a0 con 0x7fbcbc15e390 2024-03-24T10:36:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc1474640 1 -- 172.21.15.1:0/179517803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac031c30 con 0x7fbcbc15e390 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 -- 172.21.15.1:0/179517803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc15e390 msgr2=0x7fbcbc0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc15e390 0x7fbcbc0d24a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac02b7e0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 -- 172.21.15.1:0/179517803 shutdown_connections 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc0d1000 0x7fbcbc0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 --2- 172.21.15.1:0/179517803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc15e390 0x7fbcbc0d24a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 -- 172.21.15.1:0/179517803 >> 172.21.15.1:0/179517803 conn(0x7fbcbc0d3ec0 msgr2=0x7fbcbc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 -- 172.21.15.1:0/179517803 shutdown_connections 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.563+0000 7fbcc46da640 1 -- 172.21.15.1:0/179517803 wait complete. 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.564+0000 7fbcc46da640 1 Processor -- start 2024-03-24T10:36:26.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.564+0000 7fbcc46da640 1 -- start start 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcbc0b3320 0x7fbcbc0b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc15e390 0x7fbcbc15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc1f7fd0 con 0x7fbcbc0d1000 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc160ea0 con 0x7fbcbc0b3320 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc46da640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc1f3a80 con 0x7fbcbc15e390 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc1c75640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc1c75640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58562/0 (socket says 172.21.15.1:58562) 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.565+0000 7fbcc1c75640 1 -- 172.21.15.1:0/1711726991 learned_addr learned my addr 172.21.15.1:0/1711726991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 -- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc15e390 msgr2=0x7fbcbc15c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc15e390 0x7fbcbc15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 -- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcbc0b3320 msgr2=0x7fbcbc0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcbc0b3320 0x7fbcbc0b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 -- 172.21.15.1:0/1711726991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac0094d0 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc1c75640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800c7f0 tx=0x7fbcb800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8004220 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb8004380 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcbc0ccfa0 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8007860 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.566+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcbc1f3f70 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.567+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcbc04f4e0 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.567+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbcb80079c0 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.568+0000 7fbcb2ffd640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 0x7fbca006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.568+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5709+0+0 (secure 0 0 0) 0x7fbcb8089f10 con 0x7fbcbc0d1000 2024-03-24T10:36:26.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.568+0000 7fbcc2476640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 0x7fbca006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.568+0000 7fbcc2476640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 0x7fbca006dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbcac035000 tx=0x7fbcac030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.572+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbcb8054930 con 0x7fbcbc0d1000 2024-03-24T10:36:26.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.793+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbcbc0d3400 con 0x7fbcbc0d1000 2024-03-24T10:36:26.793 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-24T10:36:26.793+0000 7f388d532640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:36:26.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.795+0000 7fbcb2ffd640 1 -- 172.21.15.1:0/1711726991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fbcb801e090 con 0x7fbcbc0d1000 2024-03-24T10:36:26.795 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 msgr2=0x7fbca006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 0x7fbca006dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbcac035000 tx=0x7fbcac030000 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 msgr2=0x7fbcbc15bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800c7f0 tx=0x7fbcb800ccb0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 shutdown_connections 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fbca006b810 0x7fbca006dcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fbcbc15e390 0x7fbcbc15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcbc0d1000 0x7fbcbc15bd20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 --2- 172.21.15.1:0/1711726991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcbc0b3320 0x7fbcbc0b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.797+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 >> 172.21.15.1:0/1711726991 conn(0x7fbcbc0d3ec0 msgr2=0x7fbcbc0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.798+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 shutdown_connections 2024-03-24T10:36:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.798+0000 7fbcc46da640 1 -- 172.21.15.1:0/1711726991 wait complete. 2024-03-24T10:36:26.800 INFO:tasks.ceph.mds.a.smithi001.stderr: -2962> 2024-03-24T10:35:56.600+0000 7f1a4e90f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:36:26.810 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T10:36:26.828 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-24T10:36:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 Processor -- start 2024-03-24T10:36:26.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 -- start start 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc590162600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59015eb50 con 0x7fc5900af3f0 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59015ee30 con 0x7fc5900b4fe0 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.914+0000 7fc59676c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc590162b40 con 0x7fc59015e600 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5900af3f0 0x7fc5900af7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54982/0 (socket says 172.21.15.1:54982) 2024-03-24T10:36:26.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58ffff640 1 -- 172.21.15.1:0/3146637380 learned_addr learned my addr 172.21.15.1:0/3146637380 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196934920 0 0) 0x7fc59015eb50 con 0x7fc5900af3f0 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58f7fe640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc590162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc578003520 con 0x7fc5900af3f0 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3961891889 0 0) 0x7fc578003520 con 0x7fc5900af3f0 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 msgr2=0x7fc590162600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc590162600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 >> v1:172.21.15.3:6789/0 conn(0x7fc5900b4fe0 legacy=0x7fc5900b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59015f110 con 0x7fc5900af3f0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fc580001670 con 0x7fc5900af3f0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 >> v1:172.21.15.1:6789/0 conn(0x7fc5900af3f0 legacy=0x7fc5900af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5780039a0 0x7fc578003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc5780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc578004d00 0x7fc578005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc57800ba30 con 0x7fc5780039a0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc57800bb70 con 0x7fc578004d00 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc57800bd30 con 0x7fc59015e600 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5780039a0 0x7fc578003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc58f7fe640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc5780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.915+0000 7fc594d09640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc578004d00 0x7fc578005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 -- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 msgr2=0x7fc5780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc59015e600 0x7fc5780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 -- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc578004d00 msgr2=0x7fc578005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc578004d00 0x7fc578005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 -- 172.21.15.1:0/3146637380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc578003520 con 0x7fc5780039a0 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5780039a0 0x7fc578003d80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c210 tx=0x7fc5780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2599a0ef7bc3bd99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58002cae0 con 0x7fc5780039a0 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.916+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58002cc60 con 0x7fc5780039a0 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc58effd640 1 -- 172.21.15.1:0/3146637380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800058d0 con 0x7fc5780039a0 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 -- 172.21.15.1:0/3146637380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5780039a0 msgr2=0x7fc578003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 --2- 172.21.15.1:0/3146637380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5780039a0 0x7fc578003d80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc58000c210 tx=0x7fc5780057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc594d09640 1 -- 172.21.15.1:0/3146637380 reap_dead start 2024-03-24T10:36:26.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 -- 172.21.15.1:0/3146637380 shutdown_connections 2024-03-24T10:36:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 -- 172.21.15.1:0/3146637380 >> 172.21.15.1:0/3146637380 conn(0x7fc590072410 msgr2=0x7fc59006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 -- 172.21.15.1:0/3146637380 shutdown_connections 2024-03-24T10:36:26.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.917+0000 7fc59676c640 1 -- 172.21.15.1:0/3146637380 wait complete. 2024-03-24T10:36:26.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.918+0000 7fc59676c640 1 Processor -- start 2024-03-24T10:36:26.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.918+0000 7fc59676c640 1 -- start start 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5780039a0 0x7fc59015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc590141d30 0x7fc5901421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc590156650 con 0x7fc5900b5430 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc5900b5dd0 con 0x7fc590141d30 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc59676c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc590143b20 con 0x7fc5780039a0 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58600/0 (socket says 172.21.15.1:58600) 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 -- 172.21.15.1:0/3891676891 learned_addr learned my addr 172.21.15.1:0/3891676891 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5780039a0 0x7fc59015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc594d09640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc590141d30 0x7fc5901421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 -- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5780039a0 msgr2=0x7fc59015b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5780039a0 0x7fc59015b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 -- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc590141d30 msgr2=0x7fc5901421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.919+0000 7fc58f7fe640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc590141d30 0x7fc5901421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc58f7fe640 1 -- 172.21.15.1:0/3891676891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc590142a80 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc58f7fe640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc5840119a0 tx=0x7fc584011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58400faf0 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58400fc50 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc590142d60 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584014440 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.920+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc590143320 con 0x7fc5900b5430 2024-03-24T10:36:26.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.921+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59004f470 con 0x7fc5900b5430 2024-03-24T10:36:26.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.921+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc58401d400 con 0x7fc5900b5430 2024-03-24T10:36:26.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.922+0000 7fc58cff9640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 0x7fc57c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:26.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.922+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5795+0+0 (secure 0 0 0) 0x7fc58408a9d0 con 0x7fc5900b5430 2024-03-24T10:36:26.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.922+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 0x7fc57c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:26.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.922+0000 7fc58ffff640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 0x7fc57c06dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc580034000 tx=0x7fc580009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:26.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:26.926+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc584055390 con 0x7fc5900b5430 2024-03-24T10:36:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.145+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc5900b6640 con 0x7fc5900b5430 2024-03-24T10:36:27.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.796+0000 7fc58cff9640 1 -- 172.21.15.1:0/3891676891 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fc584058a50 con 0x7fc5900b5430 2024-03-24T10:36:27.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 msgr2=0x7fc57c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 0x7fc57c06dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc580034000 tx=0x7fc580009030 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 msgr2=0x7fc5900b5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc5840119a0 tx=0x7fc584011e60 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 shutdown_connections 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc57c06b840 0x7fc57c06dd00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc590141d30 0x7fc5901421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5900b5430 0x7fc5900b5890 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 --2- 172.21.15.1:0/3891676891 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc5780039a0 0x7fc59015b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 >> 172.21.15.1:0/3891676891 conn(0x7fc590072410 msgr2=0x7fc59006bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 shutdown_connections 2024-03-24T10:36:27.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.800+0000 7fc59676c640 1 -- 172.21.15.1:0/3891676891 wait complete. 2024-03-24T10:36:27.814 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T10:36:27.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.966+0000 7f3d05ab5640 1 Processor -- start 2024-03-24T10:36:27.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.966+0000 7f3d05ab5640 1 -- start start 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3d05ab5640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3d0006e5b0 0x7f3d0006e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3d05ab5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 0x7f3d000b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3d05ab5640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0006eed0 con 0x7f3d000b2f50 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3d05ab5640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f3d000b34d0 con 0x7f3d0006e5b0 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3d05ab5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3d000b37b0 con 0x7f3d000b4570 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3cff7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3d0006e5b0 0x7f3d0006e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3cfeffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 0x7f3d000b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3cfffff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3d000b2f50 0x7f3d0015c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:39158/0 (socket says 172.21.15.1:39158) 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3cff7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3d0006e5b0 0x7f3d0006e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:39938/0 (socket says 172.21.15.1:39938) 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.967+0000 7f3cfffff640 1 -- 172.21.15.1:0/539508212 learned_addr learned my addr 172.21.15.1:0/539508212 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:27.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.968+0000 7f3cff7fe640 1 -- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 msgr2=0x7f3d000b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.968+0000 7f3cff7fe640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 0x7f3d000b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.968+0000 7f3cff7fe640 1 -- 172.21.15.1:0/539508212 >> v1:172.21.15.3:6789/0 conn(0x7f3d000b2f50 legacy=0x7f3d0015c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.968+0000 7f3cff7fe640 1 -- 172.21.15.1:0/539508212 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d000b3a90 con 0x7f3d0006e5b0 2024-03-24T10:36:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.968+0000 7f3cfeffd640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 0x7f3d000b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3cff7fe640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0006e990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00097d0 tx=0x7f3cf002dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c296a5725638810a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/539508212 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf002f870 con 0x7f3d0006e5b0 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/539508212 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf002f9d0 con 0x7f3d0006e5b0 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3cfe7fc640 1 -- 172.21.15.1:0/539508212 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf002fd60 con 0x7f3d0006e5b0 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 -- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 msgr2=0x7f3d0006e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0006e990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00097d0 tx=0x7f3cf002dab0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 -- 172.21.15.1:0/539508212 shutdown_connections 2024-03-24T10:36:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b4570 0x7f3d000b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/539508212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0006e990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 -- 172.21.15.1:0/539508212 >> 172.21.15.1:0/539508212 conn(0x7f3d000b1710 msgr2=0x7f3d000b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 -- 172.21.15.1:0/539508212 shutdown_connections 2024-03-24T10:36:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.969+0000 7f3d05ab5640 1 -- 172.21.15.1:0/539508212 wait complete. 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.970+0000 7f3d05ab5640 1 Processor -- start 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 -- start start 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b2f50 0x7f3d000ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d00151130 con 0x7f3d000b4570 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f3d000adbc0 con 0x7f3d0006e5b0 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.971+0000 7f3d05ab5640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3d00150b70 con 0x7f3d000b2f50 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58612/0 (socket says 172.21.15.1:58612) 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 -- 172.21.15.1:0/1715996360 learned_addr learned my addr 172.21.15.1:0/1715996360 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfeffd640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b2f50 0x7f3d000ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cff7fe640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 -- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b2f50 msgr2=0x7f3d000ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b2f50 0x7f3d000ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 -- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 msgr2=0x7f3d0014e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0014e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 -- 172.21.15.1:0/1715996360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf4009400 con 0x7f3d000b4570 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3cfffff640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800cbc0 tx=0x7f3cf800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800ad40 con 0x7f3d000b4570 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf8012400 con 0x7f3d000b4570 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf0009400 con 0x7f3d000b4570 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800aea0 con 0x7f3d000b4570 2024-03-24T10:36:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.972+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d000ade60 con 0x7f3d000b4570 2024-03-24T10:36:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.973+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d0004f4e0 con 0x7f3d000b4570 2024-03-24T10:36:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.974+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3cf8007070 con 0x7f3d000b4570 2024-03-24T10:36:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.974+0000 7f3ce7fff640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 0x7f3cec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.974+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5795+0+0 (secure 0 0 0) 0x7f3cf808a0e0 con 0x7f3d000b4570 2024-03-24T10:36:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.974+0000 7f3cff7fe640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 0x7f3cec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.975+0000 7f3cff7fe640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 0x7f3cec06dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0038000 tx=0x7f3cf003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:27.978+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3cf8054ac0 con 0x7f3d000b4570 2024-03-24T10:36:28.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:28.161+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3d00144030 con 0x7f3d000b4570 2024-03-24T10:36:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.165+0000 7f3ce7fff640 1 -- 172.21.15.1:0/1715996360 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) ==== 181+0+0 (secure 0 0 0) 0x7f3cf80170d0 con 0x7f3d000b4570 2024-03-24T10:36:29.165 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.168+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 msgr2=0x7f3cec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 0x7f3cec06dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0038000 tx=0x7f3cf003b040 comp rx=0 tx=0).stop 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 msgr2=0x7f3d001500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800cbc0 tx=0x7f3cf800a430 comp rx=0 tx=0).stop 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 shutdown_connections 2024-03-24T10:36:29.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f3cec06b810 0x7f3cec06dcd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d000b4570 0x7f3d001500f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f3d000b2f50 0x7f3d000ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 --2- 172.21.15.1:0/1715996360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d0006e5b0 0x7f3d0014e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 >> 172.21.15.1:0/1715996360 conn(0x7f3d000b1710 msgr2=0x7f3d0015db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 shutdown_connections 2024-03-24T10:36:29.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.169+0000 7f3d05ab5640 1 -- 172.21.15.1:0/1715996360 wait complete. 2024-03-24T10:36:29.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T10:36:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.297+0000 7f7265b08640 1 Processor -- start 2024-03-24T10:36:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.297+0000 7f7265b08640 1 -- start start 2024-03-24T10:36:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f726015bfa0 0x7f72600d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72600cd220 0x7f72600d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72600d0720 0x7f72600ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f72600cfd90 con 0x7f726015bfa0 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f72600d0b50 con 0x7f72600d0720 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f7265b08640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f72600cdac0 con 0x7f72600cd220 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f726015bfa0 0x7f72600d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f726015bfa0 0x7f72600d2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58622/0 (socket says 172.21.15.1:58622) 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 -- 172.21.15.1:0/3148133284 learned_addr learned my addr 172.21.15.1:0/3148133284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 -- 172.21.15.1:0/3148133284 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72600cd220 msgr2=0x7f72600d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 --2- 172.21.15.1:0/3148133284 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72600cd220 0x7f72600d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 -- 172.21.15.1:0/3148133284 >> v2:172.21.15.3:3300/0 conn(0x7f72600d0720 msgr2=0x7f72600ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 --2- 172.21.15.1:0/3148133284 >> v2:172.21.15.3:3300/0 conn(0x7f72600d0720 0x7f72600ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.298+0000 7f725f7fe640 1 -- 172.21.15.1:0/3148133284 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72600cd7e0 con 0x7f726015bfa0 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.299+0000 7f725f7fe640 1 --2- 172.21.15.1:0/3148133284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726015bfa0 0x7f72600d2720 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72600716e0 tx=0x7f725002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc6f335a9f8b8df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.299+0000 7f725e7fc640 1 -- 172.21.15.1:0/3148133284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250031870 con 0x7f726015bfa0 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.299+0000 7f725e7fc640 1 -- 172.21.15.1:0/3148133284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72500319d0 con 0x7f726015bfa0 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.299+0000 7f725e7fc640 1 -- 172.21.15.1:0/3148133284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250031d60 con 0x7f726015bfa0 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 -- 172.21.15.1:0/3148133284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726015bfa0 msgr2=0x7f72600d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 --2- 172.21.15.1:0/3148133284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726015bfa0 0x7f72600d2720 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72600716e0 tx=0x7f725002b910 comp rx=0 tx=0).stop 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 -- 172.21.15.1:0/3148133284 shutdown_connections 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 --2- 172.21.15.1:0/3148133284 >> v2:172.21.15.3:3300/0 conn(0x7f72600d0720 0x7f72600ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 --2- 172.21.15.1:0/3148133284 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f72600cd220 0x7f72600d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 --2- 172.21.15.1:0/3148133284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726015bfa0 0x7f72600d2720 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 -- 172.21.15.1:0/3148133284 >> 172.21.15.1:0/3148133284 conn(0x7f72600d1c00 msgr2=0x7f72600d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 -- 172.21.15.1:0/3148133284 shutdown_connections 2024-03-24T10:36:29.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.300+0000 7f7265b08640 1 -- 172.21.15.1:0/3148133284 wait complete. 2024-03-24T10:36:29.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.301+0000 7f7265b08640 1 Processor -- start 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.301+0000 7f7265b08640 1 -- start start 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 0x7f726015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f726015bfa0 0x7f72600b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f72600b5ff0 con 0x7f72600d0720 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f72600b4f50 con 0x7f72600cd220 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f7265b08640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f72600b5230 con 0x7f726015bfa0 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58632/0 (socket says 172.21.15.1:58632) 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 -- 172.21.15.1:0/1747353373 learned_addr learned my addr 172.21.15.1:0/1747353373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725ffff640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f726015bfa0 0x7f72600b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725f7fe640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 0x7f726015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 -- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f726015bfa0 msgr2=0x7f72600b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f726015bfa0 0x7f72600b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 -- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 msgr2=0x7f726015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 0x7f726015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725effd640 1 -- 172.21.15.1:0/1747353373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72500094d0 con 0x7f72600d0720 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.302+0000 7f725f7fe640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 0x7f726015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f725effd640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f725400b4a0 tx=0x7f725400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254010610 con 0x7f72600d0720 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7254010770 con 0x7f72600d0720 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72600b5510 con 0x7f72600d0720 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254010b20 con 0x7f72600d0720 2024-03-24T10:36:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.303+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72600b5ad0 con 0x7f72600d0720 2024-03-24T10:36:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.304+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72600b5cf0 con 0x7f72600d0720 2024-03-24T10:36:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.304+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7254010c80 con 0x7f72600d0720 2024-03-24T10:36:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.305+0000 7f7247fff640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 0x7f724c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.305+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5357+0+0 (secure 0 0 0) 0x7f7254089d20 con 0x7f72600d0720 2024-03-24T10:36:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.305+0000 7f725f7fe640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 0x7f724c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.305+0000 7f725f7fe640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 0x7f724c06dcd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7250036000 tx=0x7f725003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.309+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7254054890 con 0x7f72600d0720 2024-03-24T10:36:29.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:29.494+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f726004f470 con 0x7f72600d0720 2024-03-24T10:36:30.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.167+0000 7f7247fff640 1 -- 172.21.15.1:0/1747353373 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) ==== 169+0+0 (secure 0 0 0) 0x7f725401e0d0 con 0x7f72600d0720 2024-03-24T10:36:30.167 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.170+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 msgr2=0x7f724c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.170+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 0x7f724c06dcd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7250036000 tx=0x7f725003c040 comp rx=0 tx=0).stop 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.170+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 msgr2=0x7f7260151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.170+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f725400b4a0 tx=0x7f725400b960 comp rx=0 tx=0).stop 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 shutdown_connections 2024-03-24T10:36:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f724c06b810 0x7f724c06dcd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f726015bfa0 0x7f72600b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72600d0720 0x7f7260151200 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 --2- 172.21.15.1:0/1747353373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72600cd220 0x7f726015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 >> 172.21.15.1:0/1747353373 conn(0x7f72600d1c00 msgr2=0x7f72600cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 shutdown_connections 2024-03-24T10:36:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.171+0000 7f7265b08640 1 -- 172.21.15.1:0/1747353373 wait complete. 2024-03-24T10:36:30.186 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T10:36:30.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.296+0000 7fc705bc3640 1 Processor -- start 2024-03-24T10:36:30.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.296+0000 7fc705bc3640 1 -- start start 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc705bc3640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7000b4e40 0x7fc7000d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc705bc3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc700162610 0x7fc7000d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc705bc3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7000c7a40 con 0x7fc7001615f0 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc705bc3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc7000cfd50 con 0x7fc700162610 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc705bc3640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fc7000b55b0 con 0x7fc7000b4e40 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc6fffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc700162610 0x7fc7000d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc6fffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc700162610 0x7fc7000d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58646/0 (socket says 172.21.15.1:58646) 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.297+0000 7fc6fffff640 1 -- 172.21.15.1:0/722613336 learned_addr learned my addr 172.21.15.1:0/722613336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6feffd640 1 --2- 172.21.15.1:0/722613336 >> v2:172.21.15.3:3300/0 conn(0x7fc7000b4e40 0x7fc7000d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6fffff640 1 -- 172.21.15.1:0/722613336 >> v2:172.21.15.3:3300/0 conn(0x7fc7000b4e40 msgr2=0x7fc7000d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6fffff640 1 --2- 172.21.15.1:0/722613336 >> v2:172.21.15.3:3300/0 conn(0x7fc7000b4e40 0x7fc7000d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6fffff640 1 -- 172.21.15.1:0/722613336 >> v1:172.21.15.3:6789/0 conn(0x7fc7001615f0 legacy=0x7fc700161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6fffff640 1 -- 172.21.15.1:0/722613336 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7000b52d0 con 0x7fc700162610 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.298+0000 7fc6fffff640 1 --2- 172.21.15.1:0/722613336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc700162610 0x7fc7000d1c70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc7000737b0 tx=0x7fc6f802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ede452d49a6fb07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc6fe7fc640 1 -- 172.21.15.1:0/722613336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f802cb20 con 0x7fc700162610 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc6fe7fc640 1 -- 172.21.15.1:0/722613336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f802cc80 con 0x7fc700162610 2024-03-24T10:36:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc6fe7fc640 1 -- 172.21.15.1:0/722613336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80343f0 con 0x7fc700162610 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 -- 172.21.15.1:0/722613336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc700162610 msgr2=0x7fc7000d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 --2- 172.21.15.1:0/722613336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc700162610 0x7fc7000d1c70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc7000737b0 tx=0x7fc6f802bf80 comp rx=0 tx=0).stop 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 -- 172.21.15.1:0/722613336 shutdown_connections 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 --2- 172.21.15.1:0/722613336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc700162610 0x7fc7000d1c70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 --2- 172.21.15.1:0/722613336 >> v2:172.21.15.3:3300/0 conn(0x7fc7000b4e40 0x7fc7000d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 -- 172.21.15.1:0/722613336 >> 172.21.15.1:0/722613336 conn(0x7fc7000d3ef0 msgr2=0x7fc7000d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 -- 172.21.15.1:0/722613336 shutdown_connections 2024-03-24T10:36:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.299+0000 7fc705bc3640 1 -- 172.21.15.1:0/722613336 wait complete. 2024-03-24T10:36:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.300+0000 7fc705bc3640 1 Processor -- start 2024-03-24T10:36:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 -- start start 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc700162610 0x7fc7000b9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000d1120 0x7fc7000b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000d1890 con 0x7fc7001615f0 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000b8230 con 0x7fc7000d1120 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc705bc3640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fc7000b7940 con 0x7fc700162610 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc6ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.301+0000 7fc6ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58654/0 (socket says 172.21.15.1:58654) 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 -- 172.21.15.1:0/3753454845 learned_addr learned my addr 172.21.15.1:0/3753454845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6fffff640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000d1120 0x7fc7000b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6feffd640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc700162610 0x7fc7000b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 -- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc700162610 msgr2=0x7fc7000b9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc700162610 0x7fc7000b9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 -- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000d1120 msgr2=0x7fc7000b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000d1120 0x7fc7000b86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 -- 172.21.15.1:0/3753454845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f8009b30 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6ff7fe640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc6f000c400 tx=0x7fc6f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f00052d0 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f0005450 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7000d15b0 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0007260 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.302+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7000b7ec0 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.303+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70004f470 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.304+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc6f00073c0 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.304+0000 7fc6e7fff640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 0x7fc6ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.304+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fc6f00892f0 con 0x7fc7001615f0 2024-03-24T10:36:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.304+0000 7fc6feffd640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 0x7fc6ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.304+0000 7fc6feffd640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 0x7fc6ec06dcd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc70015d630 tx=0x7fc6f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.308+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6f0054010 con 0x7fc7001615f0 2024-03-24T10:36:30.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.488+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fc7000b8050 con 0x7fc7001615f0 2024-03-24T10:36:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.491+0000 7fc6e7fff640 1 -- 172.21.15.1:0/3753454845 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7fc6f00576d0 con 0x7fc7001615f0 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 msgr2=0x7fc6ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 0x7fc6ec06dcd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc70015d630 tx=0x7fc6f4008040 comp rx=0 tx=0).stop 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 msgr2=0x7fc7000cc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc6f000c400 tx=0x7fc6f000c8c0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 shutdown_connections 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fc6ec06b810 0x7fc6ec06dcd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000d1120 0x7fc7000b86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fc700162610 0x7fc7000b9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 --2- 172.21.15.1:0/3753454845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7001615f0 0x7fc7000cc720 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 >> 172.21.15.1:0/3753454845 conn(0x7fc7000d3ef0 msgr2=0x7fc7000d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 shutdown_connections 2024-03-24T10:36:30.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.493+0000 7fc705bc3640 1 -- 172.21.15.1:0/3753454845 wait complete. 2024-03-24T10:36:30.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-24T10:36:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 Processor -- start 2024-03-24T10:36:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 -- start start 2024-03-24T10:36:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80f415f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f40b37c0 con 0x7f80f4071ea0 2024-03-24T10:36:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f4071120 con 0x7f80f4071ac0 2024-03-24T10:36:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.606+0000 7f80f9695640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f80f40b34c0 con 0x7f80f406cc00 2024-03-24T10:36:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80f4071ea0 0x7f80f40b4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42686/0 (socket says 172.21.15.1:42686) 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f37fe640 1 -- 172.21.15.1:0/3642523034 learned_addr learned my addr 172.21.15.1:0/3642523034 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f27fc640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80f415f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781163017 0 0) 0x7f80f40b37c0 con 0x7f80f4071ea0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80dc003520 con 0x7f80f4071ea0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1761178550 0 0) 0x7f80dc003520 con 0x7f80f4071ea0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 msgr2=0x7f80f415f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80f415f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 >> v1:172.21.15.3:6789/0 conn(0x7f80f4071ac0 legacy=0x7f80f40b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f415f6b0 con 0x7f80f4071ea0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f80ec0019f0 con 0x7f80f4071ea0 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 >> v1:172.21.15.1:6789/0 conn(0x7f80f4071ea0 legacy=0x7f80f40b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80dc004d00 0x7f80dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80f40b34c0 con 0x7f80dc0039a0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc00bab0 con 0x7f80dc004d00 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f80dc00de80 con 0x7f80f406cc00 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f27fc640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.607+0000 7f80f37fe640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80dc004d00 0x7f80dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 -- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 msgr2=0x7f80dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f406cc00 0x7f80dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 -- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80dc004d00 msgr2=0x7f80dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80dc004d00 0x7f80dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 -- 172.21.15.1:0/3642523034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80dc003520 con 0x7f80dc0039a0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80dc003dc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f80e4001750 tx=0x7f80dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71325d46fe4c07e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e40052d0 con 0x7f80dc0039a0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e402e050 con 0x7f80dc0039a0 2024-03-24T10:36:30.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.608+0000 7f80f1ffb640 1 -- 172.21.15.1:0/3642523034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e40035f0 con 0x7f80dc0039a0 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 -- 172.21.15.1:0/3642523034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 msgr2=0x7f80dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 --2- 172.21.15.1:0/3642523034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80dc003dc0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f80e4001750 tx=0x7f80dc0057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f37fe640 1 -- 172.21.15.1:0/3642523034 reap_dead start 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 -- 172.21.15.1:0/3642523034 shutdown_connections 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 -- 172.21.15.1:0/3642523034 >> 172.21.15.1:0/3642523034 conn(0x7f80f40b1710 msgr2=0x7f80f40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 -- 172.21.15.1:0/3642523034 shutdown_connections 2024-03-24T10:36:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.609+0000 7f80f9695640 1 -- 172.21.15.1:0/3642523034 wait complete. 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.610+0000 7f80f9695640 1 Processor -- start 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.610+0000 7f80f9695640 1 -- start start 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 0x7f80f41423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 0x7f80f415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80f415e8f0 con 0x7f80dc0039a0 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f80f41f3dd0 con 0x7f80f415df30 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f9695640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f80f4155250 con 0x7f80f414d7f0 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35172/0 (socket says 172.21.15.1:35172) 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 -- 172.21.15.1:0/948294430 learned_addr learned my addr 172.21.15.1:0/948294430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f37fe640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 0x7f80f415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 -- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 msgr2=0x7f80f41423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f27fc640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 0x7f80f41423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 0x7f80f41423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 -- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 msgr2=0x7f80f415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 0x7f80f415e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f2ffd640 1 -- 172.21.15.1:0/948294430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f4152da0 con 0x7f80dc0039a0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f27fc640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 0x7f80f41423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.611+0000 7f80f37fe640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 0x7f80f415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80f2ffd640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f80e40017b0 tx=0x7f80e4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4007a30 con 0x7f80dc0039a0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e40069e0 con 0x7f80dc0039a0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f41f3940 con 0x7f80dc0039a0 2024-03-24T10:36:30.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4006d70 con 0x7f80dc0039a0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.612+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f41f32f0 con 0x7f80dc0039a0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.613+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f40b0280 con 0x7f80dc0039a0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.613+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f80e40035f0 con 0x7f80dc0039a0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.614+0000 7f80db7fe640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 0x7f80e006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.614+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f80e40abe30 con 0x7f80dc0039a0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.614+0000 7f80f27fc640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 0x7f80e006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:30.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.614+0000 7f80f27fc640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 0x7f80e006dd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80f41f3f70 tx=0x7f80e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:30.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.618+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80e4076d00 con 0x7f80dc0039a0 2024-03-24T10:36:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:30.868+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7f80f404f4e0 con 0x7f80dc0039a0 2024-03-24T10:36:31.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.173+0000 7f80db7fe640 1 -- 172.21.15.1:0/948294430 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v170) ==== 134+0+0 (secure 0 0 0) 0x7f80e404b020 con 0x7f80dc0039a0 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.175+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 msgr2=0x7f80e006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.175+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 0x7f80e006dd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80f41f3f70 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.175+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 msgr2=0x7f80f4152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.175+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f80e40017b0 tx=0x7f80e4001120 comp rx=0 tx=0).stop 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 shutdown_connections 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f80e006b840 0x7f80e006dd00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80f415df30 0x7f80f415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f80f414d7f0 0x7f80f41423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 --2- 172.21.15.1:0/948294430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80dc0039a0 0x7f80f4152860 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 >> 172.21.15.1:0/948294430 conn(0x7f80f40b1710 msgr2=0x7f80f414e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 shutdown_connections 2024-03-24T10:36:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:31.176+0000 7f80f9695640 1 -- 172.21.15.1:0/948294430 wait complete. 2024-03-24T10:36:31.188 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner: 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-24T10:36:31.189 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:Ran 2 tests in 127.394s 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner: 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:36:31.190 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-24T10:36:31.191 INFO:tasks.cephfs_test_runner: 2024-03-24T10:36:31.192 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:36:31.455 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bbf6487d0e7c4f76a436a5205f5b71c9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-24T10:36:31.457 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-24T10:36:31.466 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-24T10:36:31.511 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-24T10:36:31.511 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:31.511 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:31.531 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:36:31.532 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-24T10:36:31.532 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:31.589 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057145,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=477 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T10:36:31.589 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:36:31.589 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:31.608 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:36:31.609 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-24T10:36:31.609 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:36:31.668 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=576 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T10:36:31.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.669 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-24T10:36:31.685 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T10:36:31.685 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:36:31.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T10:36:31.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:36:31.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-24T10:36:31.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.919 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-03-24T10:36:31.936 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-24T10:36:31.937 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T10:36:31.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:31.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T10:36:32.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:32.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T10:36:32.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:36:32.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-03-24T10:36:32.175 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-24T10:36:32.185 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T10:36:32.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:36:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.294+0000 7fa32f42a640 1 Processor -- start 2024-03-24T10:36:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 -- start start 2024-03-24T10:36:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 0x7fa328072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 0x7fa32806cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32815b9c0 con 0x7fa3281549e0 2024-03-24T10:36:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa3280d0690 con 0x7fa32806c910 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.295+0000 7fa32f42a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa32806d2a0 con 0x7fa32815b470 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32d1c6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa3281549e0 0x7fa328071a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42692/0 (socket says 172.21.15.1:42692) 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32d1c6640 1 -- 172.21.15.1:0/1483909526 learned_addr learned my addr 172.21.15.1:0/1483909526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32c9c5640 1 --2- 172.21.15.1:0/1483909526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 0x7fa328072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32d9c7640 1 --2- 172.21.15.1:0/1483909526 >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 0x7fa32806cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995207042 0 0) 0x7fa32815b9c0 con 0x7fa3281549e0 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa314003520 con 0x7fa3281549e0 2024-03-24T10:36:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3731347178 0 0) 0x7fa314003520 con 0x7fa3281549e0 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32c9c5640 1 -- 172.21.15.1:0/1483909526 >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 msgr2=0x7fa32806cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32c9c5640 1 --2- 172.21.15.1:0/1483909526 >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 0x7fa32806cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32c9c5640 1 -- 172.21.15.1:0/1483909526 >> v1:172.21.15.1:6789/0 conn(0x7fa3281549e0 legacy=0x7fa328071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32c9c5640 1 -- 172.21.15.1:0/1483909526 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa328070ca0 con 0x7fa32815b470 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.296+0000 7fa32d9c7640 1 --2- 172.21.15.1:0/1483909526 >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 0x7fa32806cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.297+0000 7fa32c9c5640 1 --2- 172.21.15.1:0/1483909526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 0x7fa328072990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa3180097a0 tx=0x7fa31802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=870238c329b28765 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.297+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324001670 con 0x7fa32815b470 2024-03-24T10:36:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.297+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa31802e050 con 0x7fa32815b470 2024-03-24T10:36:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.297+0000 7fa31ffff640 1 -- 172.21.15.1:0/1483909526 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318030710 con 0x7fa32815b470 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 -- 172.21.15.1:0/1483909526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 msgr2=0x7fa328072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 --2- 172.21.15.1:0/1483909526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 0x7fa328072990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa3180097a0 tx=0x7fa31802b790 comp rx=0 tx=0).stop 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 -- 172.21.15.1:0/1483909526 shutdown_connections 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 --2- 172.21.15.1:0/1483909526 >> v2:172.21.15.3:3300/0 conn(0x7fa32806c910 0x7fa32806cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 --2- 172.21.15.1:0/1483909526 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32815b470 0x7fa328072990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 -- 172.21.15.1:0/1483909526 >> 172.21.15.1:0/1483909526 conn(0x7fa3280b17a0 msgr2=0x7fa3280b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 -- 172.21.15.1:0/1483909526 shutdown_connections 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.298+0000 7fa32f42a640 1 -- 172.21.15.1:0/1483909526 wait complete. 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.299+0000 7fa32f42a640 1 Processor -- start 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.299+0000 7fa32f42a640 1 -- start start 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32806c910 0x7fa328153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3281549e0 0x7fa3280b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa32814ad90 con 0x7fa32815b470 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa328154240 con 0x7fa3281549e0 2024-03-24T10:36:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa3280b07a0 con 0x7fa32806c910 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35188/0 (socket says 172.21.15.1:35188) 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 -- 172.21.15.1:0/3307163632 learned_addr learned my addr 172.21.15.1:0/3307163632 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32c9c5640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3281549e0 0x7fa3280b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d1c6640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32806c910 0x7fa328153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 -- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32806c910 msgr2=0x7fa328153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32806c910 0x7fa328153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 -- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3281549e0 msgr2=0x7fa3280b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3281549e0 0x7fa3280b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 -- 172.21.15.1:0/3307163632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa320009a70 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32d9c7640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa32000efc0 tx=0x7fa320009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320009d70 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa32000f040 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.300+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa318009400 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.301+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320003990 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.301+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3280b0d60 con 0x7fa32815b470 2024-03-24T10:36:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.302+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa32804f4e0 con 0x7fa32815b470 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.302+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa3200144a0 con 0x7fa32815b470 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.302+0000 7fa31dffb640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 0x7fa30c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.303+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fa32008aaa0 con 0x7fa32815b470 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.303+0000 7fa32d1c6640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 0x7fa30c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.303+0000 7fa32d1c6640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 0x7fa30c06dd00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa324006b50 tx=0x7fa324001dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.307+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa3200557e0 con 0x7fa32815b470 2024-03-24T10:36:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.484+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa328072ed0 con 0x7fa30c06b840 2024-03-24T10:36:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.485+0000 7fa31dffb640 1 -- 172.21.15.1:0/3307163632 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7fa32400f950 con 0x7fa30c06b840 2024-03-24T10:36:32.485 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:36:32.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 msgr2=0x7fa30c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 0x7fa30c06dd00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa324006b50 tx=0x7fa324001dc0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 msgr2=0x7fa3280b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa32000efc0 tx=0x7fa320009f90 comp rx=0 tx=0).stop 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 shutdown_connections 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa30c06b840 0x7fa30c06dd00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa32815b470 0x7fa3280b4760 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3281549e0 0x7fa3280b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 --2- 172.21.15.1:0/3307163632 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa32806c910 0x7fa328153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 >> 172.21.15.1:0/3307163632 conn(0x7fa3280b17a0 msgr2=0x7fa328159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 shutdown_connections 2024-03-24T10:36:32.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.487+0000 7fa32f42a640 1 -- 172.21.15.1:0/3307163632 wait complete. 2024-03-24T10:36:32.498 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-24T10:36:30.546066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238672,"kb_used_data":5888,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736048,"statfs":{"total":773094113280,"available":772849713152,"internally_reserved":0,"allocated":6029312,"data_stored":3186330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001244"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-24T10:35:52.995039+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:35:52.995039+0000","last_peered":"2024-03-24T10:35:52.995039+0000","last_clean":"2024-03-24T10:35:52.995039+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:35:52.995039+0000","last_undegraded":"2024-03-24T10:35:52.995039+0000","last_fullsized":"2024-03-24T10:35:52.995039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1196032,"data_stored":837458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":246966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":247548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":622592,"data_stored":261123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31780,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":618496,"data_stored":261101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":602112,"data_stored":247126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32344,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339496,"statfs":{"total":96636764160,"available":96603643904,"internally_reserved":0,"allocated":1196032,"data_stored":837390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:36:32.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:36:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 Processor -- start 2024-03-24T10:36:32.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 -- start start 2024-03-24T10:36:32.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 0x7fb4940d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb4940d1000 0x7fb4940d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4940b3320 0x7fb4940cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb4940b3190 con 0x7fb4940b3320 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fb49415e770 con 0x7fb49415e390 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.602+0000 7fb49c56c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb4940d4e40 con 0x7fb4940d1000 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4940b3320 0x7fb4940cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4940b3320 0x7fb4940cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35194/0 (socket says 172.21.15.1:35194) 2024-03-24T10:36:32.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 -- 172.21.15.1:0/1197020624 learned_addr learned my addr 172.21.15.1:0/1197020624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49a308640 1 --2- 172.21.15.1:0/1197020624 >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 0x7fb4940d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb499b07640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb4940d1000 0x7fb4940d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 -- 172.21.15.1:0/1197020624 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb4940d1000 msgr2=0x7fb4940d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb4940d1000 0x7fb4940d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 -- 172.21.15.1:0/1197020624 >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 msgr2=0x7fb4940d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 --2- 172.21.15.1:0/1197020624 >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 0x7fb4940d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49ab09640 1 -- 172.21.15.1:0/1197020624 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4940d1e40 con 0x7fb4940b3320 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.603+0000 7fb49a308640 1 --2- 172.21.15.1:0/1197020624 >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 0x7fb4940d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:32.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb49ab09640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940b3320 0x7fb4940cea60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb4940d3040 tx=0x7fb48c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6753b171b200118a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb499306640 1 -- 172.21.15.1:0/1197020624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c02c950 con 0x7fb4940b3320 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb499306640 1 -- 172.21.15.1:0/1197020624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c02cab0 con 0x7fb4940b3320 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb499306640 1 -- 172.21.15.1:0/1197020624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c02ce40 con 0x7fb4940b3320 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb49c56c640 1 -- 172.21.15.1:0/1197020624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940b3320 msgr2=0x7fb4940cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.604+0000 7fb49c56c640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940b3320 0x7fb4940cea60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fb4940d3040 tx=0x7fb48c02bdb0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 -- 172.21.15.1:0/1197020624 shutdown_connections 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940b3320 0x7fb4940cea60 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 --2- 172.21.15.1:0/1197020624 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb4940d1000 0x7fb4940d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 --2- 172.21.15.1:0/1197020624 >> v2:172.21.15.3:3300/0 conn(0x7fb49415e390 0x7fb4940d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 -- 172.21.15.1:0/1197020624 >> 172.21.15.1:0/1197020624 conn(0x7fb4940d3ec0 msgr2=0x7fb4940d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 -- 172.21.15.1:0/1197020624 shutdown_connections 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.605+0000 7fb49c56c640 1 -- 172.21.15.1:0/1197020624 wait complete. 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 Processor -- start 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 -- start start 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 0x7fb49409dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 0x7fb494159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb4940cd330 con 0x7fb4940d1000 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb49409e2a0 con 0x7fb4940b3320 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.606+0000 7fb49c56c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb4940cc030 con 0x7fb49415e390 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35202/0 (socket says 172.21.15.1:35202) 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 -- 172.21.15.1:0/2109650292 learned_addr learned my addr 172.21.15.1:0/2109650292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49a308640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 0x7fb49409dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49ab09640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 0x7fb494159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 -- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 msgr2=0x7fb494159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 0x7fb494159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 -- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 msgr2=0x7fb49409dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 0x7fb49409dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 -- 172.21.15.1:0/2109650292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48c009b30 con 0x7fb4940d1000 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49ab09640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 0x7fb494159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49a308640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 0x7fb49409dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb499b07640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb48400e860 tx=0x7fb48400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4840095f0 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48400c940 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4940cc310 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48400ccd0 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.607+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4940cc8d0 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.608+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4940cca60 con 0x7fb4940d1000 2024-03-24T10:36:32.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.609+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb484013470 con 0x7fb4940d1000 2024-03-24T10:36:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.609+0000 7fb48affd640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 0x7fb47806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.609+0000 7fb49a308640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 0x7fb47806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.609+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fb484089da0 con 0x7fb4940d1000 2024-03-24T10:36:32.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.610+0000 7fb49a308640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 0x7fb47806dd00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.614+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb484054ac0 con 0x7fb4940d1000 2024-03-24T10:36:32.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.793+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4940cefa0 con 0x7fb47806b840 2024-03-24T10:36:32.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.794+0000 7fb48affd640 1 -- 172.21.15.1:0/2109650292 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13477 (secure 0 0 0) 0x7fb4900071e0 con 0x7fb47806b840 2024-03-24T10:36:32.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:36:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 msgr2=0x7fb47806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 0x7fb47806dd00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490008040 comp rx=0 tx=0).stop 2024-03-24T10:36:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 msgr2=0x7fb4940ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb48400e860 tx=0x7fb48400ed20 comp rx=0 tx=0).stop 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 shutdown_connections 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb47806b840 0x7fb47806dd00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb49415e390 0x7fb494159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4940d1000 0x7fb4940ccdf0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 --2- 172.21.15.1:0/2109650292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4940b3320 0x7fb49409dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 >> 172.21.15.1:0/2109650292 conn(0x7fb4940d3ec0 msgr2=0x7fb49406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 shutdown_connections 2024-03-24T10:36:32.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.796+0000 7fb49c56c640 1 -- 172.21.15.1:0/2109650292 wait complete. 2024-03-24T10:36:32.807 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-03-24T10:36:32.546313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238916,"kb_used_data":6104,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735804,"statfs":{"total":773094113280,"available":772849463296,"internally_reserved":0,"allocated":6250496,"data_stored":3280961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001145"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":231,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T10:36:31.128230+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:36:31.128230+0000","last_peered":"2024-03-24T10:36:31.128230+0000","last_clean":"2024-03-24T10:36:31.128230+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:36:31.128230+0000","last_undegraded":"2024-03-24T10:36:31.128230+0000","last_fullsized":"2024-03-24T10:36:31.128230+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1216512,"data_stored":849418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":259050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":849418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:36:32.807 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T10:36:32.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:36:32.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.908+0000 7f58141a1640 1 Processor -- start 2024-03-24T10:36:32.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.908+0000 7f58141a1640 1 -- start start 2024-03-24T10:36:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f58141a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f580c15e390 0x7f580c0d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f58141a1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f58141a1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c15e770 con 0x7f580c0b3320 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f58141a1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f580c0b3190 con 0x7f580c15e390 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f58141a1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f580c0d4e40 con 0x7f580c0d1000 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f5811f3d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f580c15e390 0x7f580c0d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f5811f3d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f580c15e390 0x7f580c0d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35212/0 (socket says 172.21.15.1:35212) 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f581273e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f580c0b3320 0x7f580c0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42700/0 (socket says 172.21.15.1:42700) 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.909+0000 7f5811f3d640 1 -- 172.21.15.1:0/3866407676 learned_addr learned my addr 172.21.15.1:0/3866407676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f581173c640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5811f3d640 1 -- 172.21.15.1:0/3866407676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 msgr2=0x7f580c0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5811f3d640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5811f3d640 1 -- 172.21.15.1:0/3866407676 >> v1:172.21.15.1:6789/0 conn(0x7f580c0b3320 legacy=0x7f580c0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5811f3d640 1 -- 172.21.15.1:0/3866407676 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c0d1e40 con 0x7f580c15e390 2024-03-24T10:36:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5810f3b640 1 -- 172.21.15.1:0/3866407676 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2277340209 0 0) 0x7f5804001110 con 0x7f580c0b3320 2024-03-24T10:36:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5811f3d640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c15e390 0x7f580c0d24a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009e50 tx=0x7f57fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc1f4cab05619d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.910+0000 7f5810f3b640 1 -- 172.21.15.1:0/3866407676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc02c9e0 con 0x7f580c15e390 2024-03-24T10:36:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f5810f3b640 1 -- 172.21.15.1:0/3866407676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc02cb40 con 0x7f580c15e390 2024-03-24T10:36:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f5810f3b640 1 -- 172.21.15.1:0/3866407676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0343f0 con 0x7f580c15e390 2024-03-24T10:36:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 -- 172.21.15.1:0/3866407676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c15e390 msgr2=0x7f580c0d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c15e390 0x7f580c0d24a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009e50 tx=0x7f57fc02be40 comp rx=0 tx=0).stop 2024-03-24T10:36:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 -- 172.21.15.1:0/3866407676 shutdown_connections 2024-03-24T10:36:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c0d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 --2- 172.21.15.1:0/3866407676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c15e390 0x7f580c0d24a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 -- 172.21.15.1:0/3866407676 >> 172.21.15.1:0/3866407676 conn(0x7f580c0d3ec0 msgr2=0x7f580c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 -- 172.21.15.1:0/3866407676 shutdown_connections 2024-03-24T10:36:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.911+0000 7f58141a1640 1 -- 172.21.15.1:0/3866407676 wait complete. 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.912+0000 7f58141a1640 1 Processor -- start 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.912+0000 7f58141a1640 1 -- start start 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c15e390 0x7f580c156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f580c1616f0 con 0x7f580c0b3320 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f580c15fc30 con 0x7f580c15e390 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f58141a1640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f580c15f670 con 0x7f580c0d1000 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35226/0 (socket says 172.21.15.1:35226) 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 -- 172.21.15.1:0/3734277251 learned_addr learned my addr 172.21.15.1:0/3734277251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 -- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 msgr2=0x7f580c1565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 -- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c15e390 msgr2=0x7f580c156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c15e390 0x7f580c156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.913+0000 7f5811f3d640 1 -- 172.21.15.1:0/3734277251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc009b30 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f5811f3d640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f57fc000f80 tx=0x7f57fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc03e450 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc03e5f0 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580c15f950 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc03e9a0 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.914+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580c161b10 con 0x7f580c0b3320 2024-03-24T10:36:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.915+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580c04f4e0 con 0x7f580c0b3320 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.915+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57fc03eb00 con 0x7f580c0b3320 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.916+0000 7f580a7fc640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 0x7f57f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.916+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f57fc0acdf0 con 0x7f580c0b3320 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.916+0000 7f581173c640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 0x7f57f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.916+0000 7f581173c640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 0x7f57f806dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f580c06cd90 tx=0x7f5800008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:32.920+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f57fc077c30 con 0x7f580c0b3320 2024-03-24T10:36:33.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.099+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f580c0b4ba0 con 0x7f57f806b810 2024-03-24T10:36:33.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.099+0000 7f580a7fc640 1 -- 172.21.15.1:0/3734277251 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13477 (secure 0 0 0) 0x7f580000a4c0 con 0x7f57f806b810 2024-03-24T10:36:33.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:33.100 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 msgr2=0x7f57f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 0x7f57f806dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f580c06cd90 tx=0x7f5800008040 comp rx=0 tx=0).stop 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 msgr2=0x7f580c0cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f57fc000f80 tx=0x7f57fc004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 shutdown_connections 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f57f806b810 0x7f57f806dcd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c15e390 0x7f580c156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f580c0d1000 0x7f580c1565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 --2- 172.21.15.1:0/3734277251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f580c0b3320 0x7f580c0cec90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 >> 172.21.15.1:0/3734277251 conn(0x7f580c0d3ec0 msgr2=0x7f580c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 shutdown_connections 2024-03-24T10:36:33.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.101+0000 7f58141a1640 1 -- 172.21.15.1:0/3734277251 wait complete. 2024-03-24T10:36:33.112 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-03-24T10:36:32.546313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238916,"kb_used_data":6104,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735804,"statfs":{"total":773094113280,"available":772849463296,"internally_reserved":0,"allocated":6250496,"data_stored":3280961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001145"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":231,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T10:36:31.128230+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:36:31.128230+0000","last_peered":"2024-03-24T10:36:31.128230+0000","last_clean":"2024-03-24T10:36:31.128230+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:36:31.128230+0000","last_undegraded":"2024-03-24T10:36:31.128230+0000","last_fullsized":"2024-03-24T10:36:31.128230+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1216512,"data_stored":849418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":259050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":849418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:36:33.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:36:33.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.214+0000 7ffb841fd640 1 Processor -- start 2024-03-24T10:36:33.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.214+0000 7ffb841fd640 1 -- start start 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.214+0000 7ffb841fd640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb7c15b470 0x7ffb7c072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb841fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb7c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb841fd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7c06d280 con 0x7ffb7c1549e0 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb841fd640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7c0d0690 con 0x7ffb7c15b470 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb841fd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c072ed0 con 0x7ffb7c06c910 2024-03-24T10:36:33.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb81f99640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb7c1549e0 0x7ffb7c071a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42704/0 (socket says 172.21.15.1:42704) 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb81f99640 1 -- 172.21.15.1:0/4067683042 learned_addr learned my addr 172.21.15.1:0/4067683042 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398811028 0 0) 0x7ffb7c06d280 con 0x7ffb7c1549e0 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb81798640 1 --2- 172.21.15.1:0/4067683042 >> v2:172.21.15.3:3300/0 conn(0x7ffb7c15b470 0x7ffb7c072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb64004630 con 0x7ffb7c1549e0 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb8279a640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb7c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 796115314 0 0) 0x7ffb64004630 con 0x7ffb7c1549e0 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 msgr2=0x7ffb7c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb7c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 >> v2:172.21.15.3:3300/0 conn(0x7ffb7c15b470 msgr2=0x7ffb7c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 --2- 172.21.15.1:0/4067683042 >> v2:172.21.15.3:3300/0 conn(0x7ffb7c15b470 0x7ffb7c072990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c15b9c0 con 0x7ffb7c1549e0 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7ffb740019f0 con 0x7ffb7c1549e0 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 >> v1:172.21.15.1:6789/0 conn(0x7ffb7c1549e0 legacy=0x7ffb7c071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb64004ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb640058d0 0x7ffb64005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb64006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c0d0690 con 0x7ffb64004ab0 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c072ed0 con 0x7ffb640058d0 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6400ca90 con 0x7ffb7c06c910 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb81f99640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb64004ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb81798640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb640058d0 0x7ffb64005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.215+0000 7ffb8279a640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb64006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 -- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 msgr2=0x7ffb64006290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb64006290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 -- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb640058d0 msgr2=0x7ffb64005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:33.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb640058d0 0x7ffb64005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 -- 172.21.15.1:0/4067683042 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64004630 con 0x7ffb64004ab0 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81798640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb640058d0 0x7ffb64005d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb8279a640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c06c910 0x7ffb64006290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.216+0000 7ffb81f99640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb64004ed0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400a410 tx=0x7ffb64006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d756f8f1c090dda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74031070 con 0x7ffb64004ab0 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb740070d0 con 0x7ffb64004ab0 2024-03-24T10:36:33.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb80f97640 1 -- 172.21.15.1:0/4067683042 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74007480 con 0x7ffb64004ab0 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 -- 172.21.15.1:0/4067683042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 msgr2=0x7ffb64004ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 --2- 172.21.15.1:0/4067683042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb64004ed0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400a410 tx=0x7ffb64006900 comp rx=0 tx=0).stop 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb8279a640 1 -- 172.21.15.1:0/4067683042 reap_dead start 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 -- 172.21.15.1:0/4067683042 shutdown_connections 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 -- 172.21.15.1:0/4067683042 >> 172.21.15.1:0/4067683042 conn(0x7ffb7c0b17a0 msgr2=0x7ffb7c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 -- 172.21.15.1:0/4067683042 shutdown_connections 2024-03-24T10:36:33.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.217+0000 7ffb841fd640 1 -- 172.21.15.1:0/4067683042 wait complete. 2024-03-24T10:36:33.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.218+0000 7ffb841fd640 1 Processor -- start 2024-03-24T10:36:33.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.218+0000 7ffb841fd640 1 -- start start 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c0b0720 0x7ffb7c0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c1f43d0 0x7ffb7c1f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c1f1990 con 0x7ffb64004ab0 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c0b10e0 con 0x7ffb7c1f43d0 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb841fd640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c0cddf0 con 0x7ffb7c0b0720 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35234/0 (socket says 172.21.15.1:35234) 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 -- 172.21.15.1:0/3253999489 learned_addr learned my addr 172.21.15.1:0/3253999489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 -- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c0b0720 msgr2=0x7ffb7c0b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c0b0720 0x7ffb7c0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 -- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c1f43d0 msgr2=0x7ffb7c1f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c1f43d0 0x7ffb7c1f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.219+0000 7ffb81f99640 1 -- 172.21.15.1:0/3253999489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c0ce0d0 con 0x7ffb64004ab0 2024-03-24T10:36:33.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb81f99640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400a920 tx=0x7ffb74001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb740019e0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb740075f0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7c0ce3b0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb740079a0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.220+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7c0cdae0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.221+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7c0ce8d0 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.221+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb74040420 con 0x7ffb64004ab0 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.222+0000 7ffb7a7fc640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 0x7ffb6806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.222+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7ffb740ac150 con 0x7ffb64004ab0 2024-03-24T10:36:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.222+0000 7ffb81798640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 0x7ffb6806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.222+0000 7ffb81798640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 0x7ffb6806dd00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c154170 tx=0x7ffb70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.226+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb74076f90 con 0x7ffb64004ab0 2024-03-24T10:36:33.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.407+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb7c04f4c0 con 0x7ffb64004ab0 2024-03-24T10:36:33.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.408+0000 7ffb7a7fc640 1 -- 172.21.15.1:0/3253999489 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+12886 (secure 0 0 0) 0x7ffb7407a650 con 0x7ffb64004ab0 2024-03-24T10:36:33.408 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:33.409 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":53,"fsid":"a0192ab9-51d5-4157-8f81-4a83111a7dfa","created":"2024-03-24T10:33:30.212857+0000","modified":"2024-03-24T10:36:30.166115+0000","last_up_change":"2024-03-24T10:33:41.000977+0000","last_in_change":"2024-03-24T10:33:32.470118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:42.558975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5b76212-d040-46a1-99aa-53cf6f2a06bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6809","nonce":2235589886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6811","nonce":2235589886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6815","nonce":2235589886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2235589886},{"type":"v1","addr":"172.21.15.1:6813","nonce":2235589886}]},"public_addr":"172.21.15.1:6809/2235589886","cluster_addr":"172.21.15.1:6811/2235589886","heartbeat_back_addr":"172.21.15.1:6815/2235589886","heartbeat_front_addr":"172.21.15.1:6813/2235589886","state":["exists","up"]},{"osd":1,"uuid":"beba3353-6c19-4121-adea-b93d378c084a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6801","nonce":3667481935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6803","nonce":3667481935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6807","nonce":3667481935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3667481935},{"type":"v1","addr":"172.21.15.1:6805","nonce":3667481935}]},"public_addr":"172.21.15.1:6801/3667481935","cluster_addr":"172.21.15.1:6803/3667481935","heartbeat_back_addr":"172.21.15.1:6807/3667481935","heartbeat_front_addr":"172.21.15.1:6805/3667481935","state":["exists","up"]},{"osd":2,"uuid":"7c577a7b-20a1-4907-b379-3674f2cb3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6818","nonce":1358018271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6821","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6823","nonce":1358018271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6831","nonce":1358018271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6825","nonce":1358018271},{"type":"v1","addr":"172.21.15.1:6827","nonce":1358018271}]},"public_addr":"172.21.15.1:6818/1358018271","cluster_addr":"172.21.15.1:6823/1358018271","heartbeat_back_addr":"172.21.15.1:6831/1358018271","heartbeat_front_addr":"172.21.15.1:6827/1358018271","state":["exists","up"]},{"osd":3,"uuid":"89097d54-de31-43a5-b6ff-3ad809e70b6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6817","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6819","nonce":2415676065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6822","nonce":2415676065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6829","nonce":2415676065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":2415676065},{"type":"v1","addr":"172.21.15.1:6826","nonce":2415676065}]},"public_addr":"172.21.15.1:6819/2415676065","cluster_addr":"172.21.15.1:6822/2415676065","heartbeat_back_addr":"172.21.15.1:6829/2415676065","heartbeat_front_addr":"172.21.15.1:6826/2415676065","state":["exists","up"]},{"osd":4,"uuid":"1b743f53-f298-412e-9fe7-69695fbe74a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6801","nonce":3439594233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6803","nonce":3439594233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6807","nonce":3439594233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3439594233},{"type":"v1","addr":"172.21.15.3:6805","nonce":3439594233}]},"public_addr":"172.21.15.3:6801/3439594233","cluster_addr":"172.21.15.3:6803/3439594233","heartbeat_back_addr":"172.21.15.3:6807/3439594233","heartbeat_front_addr":"172.21.15.3:6805/3439594233","state":["exists","up"]},{"osd":5,"uuid":"aec7a526-7963-4327-9f94-9bcc1bc8c669","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6825","nonce":3670894664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6827","nonce":3670894664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6831","nonce":3670894664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":3670894664},{"type":"v1","addr":"172.21.15.3:6829","nonce":3670894664}]},"public_addr":"172.21.15.3:6825/3670894664","cluster_addr":"172.21.15.3:6827/3670894664","heartbeat_back_addr":"172.21.15.3:6831/3670894664","heartbeat_front_addr":"172.21.15.3:6829/3670894664","state":["exists","up"]},{"osd":6,"uuid":"b38bb608-40ac-4f3f-9ece-e4d002045f3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6809","nonce":2701673417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6811","nonce":2701673417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6815","nonce":2701673417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2701673417},{"type":"v1","addr":"172.21.15.3:6813","nonce":2701673417}]},"public_addr":"172.21.15.3:6809/2701673417","cluster_addr":"172.21.15.3:6811/2701673417","heartbeat_back_addr":"172.21.15.3:6815/2701673417","heartbeat_front_addr":"172.21.15.3:6813/2701673417","state":["exists","up"]},{"osd":7,"uuid":"001d918f-3181-4d41-82e5-6e03f6410e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6817","nonce":1606588586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6819","nonce":1606588586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6823","nonce":1606588586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1606588586},{"type":"v1","addr":"172.21.15.3:6821","nonce":1606588586}]},"public_addr":"172.21.15.3:6817/1606588586","cluster_addr":"172.21.15.3:6819/1606588586","heartbeat_back_addr":"172.21.15.3:6823/1606588586","heartbeat_front_addr":"172.21.15.3:6821/1606588586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.640435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.564729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.651331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.582751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.650744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.677995+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.565342+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.618456+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6837/1119214087":"2024-03-25T10:36:26.794355+0000","172.21.15.1:6836/1119214087":"2024-03-25T10:36:26.794355+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 msgr2=0x7ffb6806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 0x7ffb6806dd00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c154170 tx=0x7ffb70008040 comp rx=0 tx=0).stop 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 msgr2=0x7ffb7c0cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400a920 tx=0x7ffb74001d80 comp rx=0 tx=0).stop 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 shutdown_connections 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7ffb6806b840 0x7ffb6806dd00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c1f43d0 0x7ffb7c1f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7ffb7c0b0720 0x7ffb7c0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 --2- 172.21.15.1:0/3253999489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb64004ab0 0x7ffb7c0cd5a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 >> 172.21.15.1:0/3253999489 conn(0x7ffb7c0b17a0 msgr2=0x7ffb7c15d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 shutdown_connections 2024-03-24T10:36:33.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:33.410+0000 7ffb841fd640 1 -- 172.21.15.1:0/3253999489 wait complete. 2024-03-24T10:36:34.422 INFO:tasks.ceph:Scrubbing osd.0 2024-03-24T10:36:34.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.540+0000 7fb2d2684640 1 Processor -- start 2024-03-24T10:36:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.540+0000 7fb2d2684640 1 -- start start 2024-03-24T10:36:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2d2684640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2cc0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2d2684640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2cc0cdac0 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2d2684640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2cc0cd7e0 con 0x7fb2cc0d08f0 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2d2684640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc0cfd90 con 0x7fb2cc0cd220 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2cbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2cc15bfa0 0x7fb2cc0d01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42708/0 (socket says 172.21.15.1:42708) 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2cbfff640 1 -- 172.21.15.1:0/4153921564 learned_addr learned my addr 172.21.15.1:0/4153921564 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945601046 0 0) 0x7fb2cc0cdac0 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b8003520 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.541+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2cc0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1506804240 0 0) 0x7fb2b8003520 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 msgr2=0x7fb2cc0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2cc0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 >> v1:172.21.15.3:6789/0 conn(0x7fb2cc0d08f0 legacy=0x7fb2cc0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc0b1210 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fb2b4001690 con 0x7fb2cc15bfa0 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 >> v1:172.21.15.1:6789/0 conn(0x7fb2cc15bfa0 legacy=0x7fb2cc0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2b80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8003f20 0x7fb2b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8004d00 0x7fb2b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0cfd90 con 0x7fb2b8004d00 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b800ba60 con 0x7fb2b8003f20 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb2b800bc20 con 0x7fb2cc0cd220 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8004d00 0x7fb2b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2cbfff640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2b80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2cb7fe640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8003f20 0x7fb2b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 -- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 msgr2=0x7fb2b80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0cd220 0x7fb2b80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 -- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8003f20 msgr2=0x7fb2b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8003f20 0x7fb2b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.542+0000 7fb2d0c21640 1 -- 172.21.15.1:0/4153921564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b8003520 con 0x7fb2b8004d00 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.543+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8004d00 0x7fb2b8005180 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009fc0 tx=0x7fb2b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744e83964c09cc8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.543+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c402cc20 con 0x7fb2b8004d00 2024-03-24T10:36:34.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.543+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c400f040 con 0x7fb2b8004d00 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.543+0000 7fb2caffd640 1 -- 172.21.15.1:0/4153921564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4034440 con 0x7fb2b8004d00 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 -- 172.21.15.1:0/4153921564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8004d00 msgr2=0x7fb2b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 --2- 172.21.15.1:0/4153921564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2b8004d00 0x7fb2b8005180 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009fc0 tx=0x7fb2b80057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d0c21640 1 -- 172.21.15.1:0/4153921564 reap_dead start 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 -- 172.21.15.1:0/4153921564 shutdown_connections 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 -- 172.21.15.1:0/4153921564 >> 172.21.15.1:0/4153921564 conn(0x7fb2cc0d1c00 msgr2=0x7fb2cc0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 -- 172.21.15.1:0/4153921564 shutdown_connections 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.544+0000 7fb2d2684640 1 -- 172.21.15.1:0/4153921564 wait complete. 2024-03-24T10:36:34.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.545+0000 7fb2d2684640 1 Processor -- start 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.545+0000 7fb2d2684640 1 -- start start 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8004d00 0x7fb2cc15e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 0x7fb2cc15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0b67a0 0x7fb2cc0b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0b7170 con 0x7fb2cc15ec60 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0b5c20 con 0x7fb2b8004d00 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d2684640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc1f28c0 con 0x7fb2cc0b67a0 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 0x7fb2cc15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 0x7fb2cc15f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35260/0 (socket says 172.21.15.1:35260) 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 -- 172.21.15.1:0/917958627 learned_addr learned my addr 172.21.15.1:0/917958627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0b67a0 0x7fb2cc0b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cbfff640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8004d00 0x7fb2cc15e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 -- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0b67a0 msgr2=0x7fb2cc0b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fb2cc0b67a0 0x7fb2cc0b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 -- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8004d00 msgr2=0x7fb2cc15e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8004d00 0x7fb2cc15e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.546+0000 7fb2cb7fe640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc0b4b80 con 0x7fb2cc15ec60 2024-03-24T10:36:34.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2cbfff640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2b8004d00 0x7fb2cc15e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2cb7fe640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 0x7fb2cc15f0e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00f7f0 tx=0x7fb2bc00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0050d0 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc005230 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc0b4e60 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00d730 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.547+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc0b5420 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.548+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2cc0b17d0 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb2bc00d890 con 0x7fb2cc15ec60 2024-03-24T10:36:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb2ac06b810 0x7fb2ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fb2bc089d60 con 0x7fb2cc15ec60 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2cbfff640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb2ac06b810 0x7fb2ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7fb2ac0716d0 0x7fb2ac073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2ac0741a0 con 0x7fb2ac0716d0 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb2bc08c420 con 0x7fb2cc15ec60 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.549+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7fb2ac0716d0 0x7fb2ac073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.550+0000 7fb2cbfff640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb2ac06b810 0x7fb2ac06dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40017c0 tx=0x7fb2b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.550+0000 7fb2d0c21640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7fb2ac0716d0 0x7fb2ac073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.550+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== osd.0 v2:172.21.15.1:6808/2235589886 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2ac0741a0 con 0x7fb2ac0716d0 2024-03-24T10:36:34.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.564+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 --> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb2cc04f470 con 0x7fb2ac0716d0 2024-03-24T10:36:34.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2c8ff9640 1 -- 172.21.15.1:0/917958627 <== osd.0 v2:172.21.15.1:6808/2235589886 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb2cc04f470 con 0x7fb2ac0716d0 2024-03-24T10:36:34.565 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:34.565 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:34.565 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:34.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7fb2ac0716d0 msgr2=0x7fb2ac073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6808/2235589886,v1:172.21.15.1:6809/2235589886] conn(0x7fb2ac0716d0 0x7fb2ac073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb2ac06b810 msgr2=0x7fb2ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fb2ac06b810 0x7fb2ac06dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40017c0 tx=0x7fb2b4009040 comp rx=0 tx=0).stop 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 msgr2=0x7fb2cc15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 --2- 172.21.15.1:0/917958627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2cc15ec60 0x7fb2cc15f0e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00f7f0 tx=0x7fb2bc00fcb0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d0c21640 1 -- 172.21.15.1:0/917958627 reap_dead start 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 shutdown_connections 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 >> 172.21.15.1:0/917958627 conn(0x7fb2cc0d1c00 msgr2=0x7fb2cc0b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 shutdown_connections 2024-03-24T10:36:34.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.565+0000 7fb2d2684640 1 -- 172.21.15.1:0/917958627 wait complete. 2024-03-24T10:36:34.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-24T10:36:34.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.679+0000 7f42c958a640 1 Processor -- start 2024-03-24T10:36:34.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.679+0000 7f42c958a640 1 -- start start 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c4154fe0 0x7f42c406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42c406b5e0 0x7f42c40adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 0x7f42c4161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f42c406d250 con 0x7f42c406b5e0 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f42c40ae640 con 0x7f42c40ae1b0 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c958a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42c41553c0 con 0x7f42c4154fe0 2024-03-24T10:36:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42c406b5e0 0x7f42c40adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f42c406b5e0 0x7f42c40adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35274/0 (socket says 172.21.15.1:35274) 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 -- 172.21.15.1:0/1969867966 learned_addr learned my addr 172.21.15.1:0/1969867966 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c2ffd640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c4154fe0 0x7f42c406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c37fe640 1 --2- 172.21.15.1:0/1969867966 >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 0x7f42c4161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 -- 172.21.15.1:0/1969867966 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c4154fe0 msgr2=0x7f42c406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c4154fe0 0x7f42c406b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 -- 172.21.15.1:0/1969867966 >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 msgr2=0x7f42c4161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 --2- 172.21.15.1:0/1969867966 >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 0x7f42c4161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42bbfff640 1 -- 172.21.15.1:0/1969867966 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c406c7e0 con 0x7f42c406b5e0 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.680+0000 7f42c37fe640 1 --2- 172.21.15.1:0/1969867966 >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 0x7f42c4161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.681+0000 7f42bbfff640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c406b5e0 0x7f42c40adc70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f42c415c000 tx=0x7f42ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217ed0cf93833023 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.681+0000 7f42c27fc640 1 -- 172.21.15.1:0/1969867966 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac031870 con 0x7f42c406b5e0 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.681+0000 7f42c27fc640 1 -- 172.21.15.1:0/1969867966 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac0319d0 con 0x7f42c406b5e0 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.681+0000 7f42c27fc640 1 -- 172.21.15.1:0/1969867966 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac031d60 con 0x7f42c406b5e0 2024-03-24T10:36:34.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 -- 172.21.15.1:0/1969867966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c406b5e0 msgr2=0x7f42c40adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c406b5e0 0x7f42c40adc70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f42c415c000 tx=0x7f42ac02b910 comp rx=0 tx=0).stop 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 -- 172.21.15.1:0/1969867966 shutdown_connections 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 --2- 172.21.15.1:0/1969867966 >> v2:172.21.15.3:3300/0 conn(0x7f42c40ae1b0 0x7f42c4161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c406b5e0 0x7f42c40adc70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 --2- 172.21.15.1:0/1969867966 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c4154fe0 0x7f42c406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 -- 172.21.15.1:0/1969867966 >> 172.21.15.1:0/1969867966 conn(0x7f42c40d0570 msgr2=0x7f42c40d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 -- 172.21.15.1:0/1969867966 shutdown_connections 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.682+0000 7f42c958a640 1 -- 172.21.15.1:0/1969867966 wait complete. 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.683+0000 7f42c958a640 1 Processor -- start 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.683+0000 7f42c958a640 1 -- start start 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c406b5e0 0x7f42c40cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42c40ae1b0 0x7f42c40cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42c415de00 con 0x7f42c4154fe0 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f42c415d7f0 con 0x7f42c40ae1b0 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f42c40cfed0 con 0x7f42c406b5e0 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35284/0 (socket says 172.21.15.1:35284) 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 -- 172.21.15.1:0/1856046053 learned_addr learned my addr 172.21.15.1:0/1856046053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c2ffd640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c406b5e0 0x7f42c40cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42bbfff640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42c40ae1b0 0x7f42c40cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 -- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c406b5e0 msgr2=0x7f42c40cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c406b5e0 0x7f42c40cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 -- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42c40ae1b0 msgr2=0x7f42c40cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42c40ae1b0 0x7f42c40cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 -- 172.21.15.1:0/1856046053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac0094d0 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c37fe640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f42bc007e10 tx=0x7f42bc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003b90 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc00f040 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c40cfbf0 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00de80 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.684+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c40ccf20 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.685+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c4069200 con 0x7f42c4154fe0 2024-03-24T10:36:34.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.686+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42bc004030 con 0x7f42c4154fe0 2024-03-24T10:36:34.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.686+0000 7f42bb7fe640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 0x7f42a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.686+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f42bc08a190 con 0x7f42c4154fe0 2024-03-24T10:36:34.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.686+0000 7f42c2ffd640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 0x7f42a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.687+0000 7f42c2ffd640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 0x7f42a406dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009a30 tx=0x7f42b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.690+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42bc054f40 con 0x7f42c4154fe0 2024-03-24T10:36:34.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.871+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f42c406beb0 con 0x7f42a406b810 2024-03-24T10:36:34.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.871+0000 7f42bb7fe640 1 -- 172.21.15.1:0/1856046053 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f42b4009680 con 0x7f42a406b810 2024-03-24T10:36:34.872 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 msgr2=0x7f42a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 0x7f42a406dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009a30 tx=0x7f42b4008040 comp rx=0 tx=0).stop 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 msgr2=0x7f42c406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f42bc007e10 tx=0x7f42bc00d4a0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 shutdown_connections 2024-03-24T10:36:34.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f42a406b810 0x7f42a406dcd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c4154fe0 0x7f42c406e660 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42c40ae1b0 0x7f42c40cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 --2- 172.21.15.1:0/1856046053 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f42c406b5e0 0x7f42c40cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 >> 172.21.15.1:0/1856046053 conn(0x7f42c40d0570 msgr2=0x7f42c41618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:34.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 shutdown_connections 2024-03-24T10:36:34.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.873+0000 7f42c958a640 1 -- 172.21.15.1:0/1856046053 wait complete. 2024-03-24T10:36:34.884 INFO:tasks.ceph:Scrubbing osd.1 2024-03-24T10:36:34.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:34.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.990+0000 7f4798839640 1 Processor -- start 2024-03-24T10:36:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.990+0000 7f4798839640 1 -- start start 2024-03-24T10:36:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4798839640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4794144b10 0x7f4794072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4798839640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f479406b7b0 0x7f47940cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4798839640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4794073050 con 0x7f479415fa30 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4798839640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f47940cf250 con 0x7f479406b7b0 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4798839640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4794145a10 con 0x7f4794144b10 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f478bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f479415fa30 0x7f479406b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42720/0 (socket says 172.21.15.1:42720) 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f478bfff640 1 -- 172.21.15.1:0/1509857631 learned_addr learned my addr 172.21.15.1:0/1509857631 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4792d9d640 1 --2- 172.21.15.1:0/1509857631 >> v2:172.21.15.3:3300/0 conn(0x7f479406b7b0 0x7f47940cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f479259c640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4794144b10 0x7f4794072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001269612 0 0) 0x7f4794073050 con 0x7f479415fa30 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4780003520 con 0x7f479415fa30 2024-03-24T10:36:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1950339357 0 0) 0x7f4780003520 con 0x7f479415fa30 2024-03-24T10:36:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4794144b10 msgr2=0x7f4794072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4794144b10 0x7f4794072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 >> v2:172.21.15.3:3300/0 conn(0x7f479406b7b0 msgr2=0x7f47940cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.991+0000 7f4791d9b640 1 --2- 172.21.15.1:0/1509857631 >> v2:172.21.15.3:3300/0 conn(0x7f479406b7b0 0x7f47940cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479415fec0 con 0x7f479415fa30 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7f477c001670 con 0x7f479415fa30 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 >> v1:172.21.15.1:6789/0 conn(0x7f479415fa30 legacy=0x7f479406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4780003970 0x7f4780003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47800047c0 0x7f4780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4780005180 0x7f478000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f47940cf250 con 0x7f47800047c0 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4794145a10 con 0x7f4780003970 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f478000bec0 con 0x7f4780005180 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f479259c640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4794144b10 0x7f4794072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4792d9d640 1 --2- 172.21.15.1:0/1509857631 >> v2:172.21.15.3:3300/0 conn(0x7f479406b7b0 0x7f47940cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47800047c0 0x7f4780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f479259c640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4780003970 0x7f4780003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f4792d9d640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4780005180 0x7f478000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 -- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4780005180 msgr2=0x7f478000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4780005180 0x7f478000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 -- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4780003970 msgr2=0x7f4780003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4780003970 0x7f4780003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.992+0000 7f478bfff640 1 -- 172.21.15.1:0/1509857631 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4780003520 con 0x7f47800047c0 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f478bfff640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47800047c0 0x7f4780004c40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c270 tx=0x7f47800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d0480b61146995 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c02cae0 con 0x7f47800047c0 2024-03-24T10:36:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f477c02cc40 con 0x7f47800047c0 2024-03-24T10:36:35.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4791d9b640 1 -- 172.21.15.1:0/1509857631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c0058d0 con 0x7f47800047c0 2024-03-24T10:36:35.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 -- 172.21.15.1:0/1509857631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47800047c0 msgr2=0x7f4780004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 --2- 172.21.15.1:0/1509857631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47800047c0 0x7f4780004c40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c270 tx=0x7f47800057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4792d9d640 1 -- 172.21.15.1:0/1509857631 reap_dead start 2024-03-24T10:36:35.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 -- 172.21.15.1:0/1509857631 shutdown_connections 2024-03-24T10:36:35.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 -- 172.21.15.1:0/1509857631 >> 172.21.15.1:0/1509857631 conn(0x7f479406c1b0 msgr2=0x7f479406c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 -- 172.21.15.1:0/1509857631 shutdown_connections 2024-03-24T10:36:35.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.993+0000 7f4798839640 1 -- 172.21.15.1:0/1509857631 wait complete. 2024-03-24T10:36:35.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 Processor -- start 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 -- start start 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f47800047c0 0x7f479415b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47940c59f0 0x7f47940c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 0x7f47940c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f479415bde0 con 0x7f47940c5e70 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f47940c74e0 con 0x7f47940c59f0 2024-03-24T10:36:35.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.995+0000 7f4798839640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f47940c6440 con 0x7f47800047c0 2024-03-24T10:36:35.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 0x7f47940c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 0x7f47940c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35302/0 (socket says 172.21.15.1:35302) 2024-03-24T10:36:35.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 -- 172.21.15.1:0/872532832 learned_addr learned my addr 172.21.15.1:0/872532832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f479259c640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f47800047c0 0x7f479415b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f478bfff640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47940c59f0 0x7f47940c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 -- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f47800047c0 msgr2=0x7f479415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f47800047c0 0x7f479415b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 -- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47940c59f0 msgr2=0x7f47940c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47940c59f0 0x7f47940c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47940c6720 con 0x7f47940c5e70 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f478bfff640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47940c59f0 0x7f47940c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4792d9d640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 0x7f47940c81b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4794072140 tx=0x7f478c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c00e070 con 0x7f47940c5e70 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f478c011e30 con 0x7f47940c5e70 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47940c6a00 con 0x7f47940c5e70 2024-03-24T10:36:35.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c003990 con 0x7f47940c5e70 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.996+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47940c6fc0 con 0x7f47940c5e70 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.997+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4794160760 con 0x7f47940c5e70 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f478c003af0 con 0x7f47940c5e70 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477406b810 0x7f477406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f478c08a660 con 0x7f47940c5e70 2024-03-24T10:36:35.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f479259c640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477406b810 0x7f477406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f47740716d0 0x7f4774073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47740741a0 con 0x7f47740716d0 2024-03-24T10:36:35.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f478c08cd20 con 0x7f47940c5e70 2024-03-24T10:36:35.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.998+0000 7f478bfff640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f47740716d0 0x7f4774073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.999+0000 7f479259c640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477406b810 0x7f477406dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f47940b3800 tx=0x7f4784008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.999+0000 7f478bfff640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f47740716d0 0x7f4774073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:34.999+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== osd.1 v2:172.21.15.1:6800/3667481935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f47740741a0 con 0x7f47740716d0 2024-03-24T10:36:35.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 --> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4794160480 con 0x7f47740716d0 2024-03-24T10:36:35.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f478affd640 1 -- 172.21.15.1:0/872532832 <== osd.1 v2:172.21.15.1:6800/3667481935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f477c02ca60 con 0x7f47740716d0 2024-03-24T10:36:35.013 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:35.014 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:35.014 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f47740716d0 msgr2=0x7f4774073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6800/3667481935,v1:172.21.15.1:6801/3667481935] conn(0x7f47740716d0 0x7f4774073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477406b810 msgr2=0x7f477406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f477406b810 0x7f477406dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f47940b3800 tx=0x7f4784008040 comp rx=0 tx=0).stop 2024-03-24T10:36:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 msgr2=0x7f47940c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 --2- 172.21.15.1:0/872532832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47940c5e70 0x7f47940c81b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4794072140 tx=0x7f478c009f90 comp rx=0 tx=0).stop 2024-03-24T10:36:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4792d9d640 1 -- 172.21.15.1:0/872532832 reap_dead start 2024-03-24T10:36:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 shutdown_connections 2024-03-24T10:36:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 >> 172.21.15.1:0/872532832 conn(0x7f479406c1b0 msgr2=0x7f479406f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 shutdown_connections 2024-03-24T10:36:35.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.013+0000 7f4798839640 1 -- 172.21.15.1:0/872532832 wait complete. 2024-03-24T10:36:35.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-24T10:36:35.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.127+0000 7f8619015640 1 Processor -- start 2024-03-24T10:36:35.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.127+0000 7f8619015640 1 -- start start 2024-03-24T10:36:35.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86140cfd00 0x7f86140cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86140cd830 0x7f86140d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140d00e0 0x7f86140b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f86140d1930 con 0x7f86140cfd00 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f86140d0be0 con 0x7f86140cd830 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8619015640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f86140cc790 con 0x7f86140d00e0 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86140cfd00 0x7f86140cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f86140cfd00 0x7f86140cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35314/0 (socket says 172.21.15.1:35314) 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 -- 172.21.15.1:0/1503354906 learned_addr learned my addr 172.21.15.1:0/1503354906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f860bfff640 1 --2- 172.21.15.1:0/1503354906 >> v2:172.21.15.3:3300/0 conn(0x7f86140cd830 0x7f86140d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f861359e640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140d00e0 0x7f86140b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 -- 172.21.15.1:0/1503354906 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140d00e0 msgr2=0x7f86140b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140d00e0 0x7f86140b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 -- 172.21.15.1:0/1503354906 >> v2:172.21.15.3:3300/0 conn(0x7f86140cd830 msgr2=0x7f86140d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1503354906 >> v2:172.21.15.3:3300/0 conn(0x7f86140cd830 0x7f86140d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.128+0000 7f8612d9d640 1 -- 172.21.15.1:0/1503354906 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86140cca70 con 0x7f86140cfd00 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.129+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140cfd00 0x7f86140cf0a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8614071b60 tx=0x7f860402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43fca1a9edfb76c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.129+0000 7f861259c640 1 -- 172.21.15.1:0/1503354906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604031870 con 0x7f86140cfd00 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.129+0000 7f861259c640 1 -- 172.21.15.1:0/1503354906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86040319d0 con 0x7f86140cfd00 2024-03-24T10:36:35.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.129+0000 7f861259c640 1 -- 172.21.15.1:0/1503354906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604031d60 con 0x7f86140cfd00 2024-03-24T10:36:35.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 -- 172.21.15.1:0/1503354906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140cfd00 msgr2=0x7f86140cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140cfd00 0x7f86140cf0a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8614071b60 tx=0x7f860402b910 comp rx=0 tx=0).stop 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 -- 172.21.15.1:0/1503354906 shutdown_connections 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140d00e0 0x7f86140b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 --2- 172.21.15.1:0/1503354906 >> v2:172.21.15.3:3300/0 conn(0x7f86140cd830 0x7f86140d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 --2- 172.21.15.1:0/1503354906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140cfd00 0x7f86140cf0a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 -- 172.21.15.1:0/1503354906 >> 172.21.15.1:0/1503354906 conn(0x7f86140d1c80 msgr2=0x7f86140d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 -- 172.21.15.1:0/1503354906 shutdown_connections 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.130+0000 7f8619015640 1 -- 172.21.15.1:0/1503354906 wait complete. 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.131+0000 7f8619015640 1 Processor -- start 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.131+0000 7f8619015640 1 -- start start 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86140cd830 0x7f861415d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 0x7f86140b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f861414c530 con 0x7f86140d00e0 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f8614155b30 con 0x7f86140cd830 2024-03-24T10:36:35.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f861415e7a0 con 0x7f86140cfd00 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35318/0 (socket says 172.21.15.1:35318) 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 -- 172.21.15.1:0/1310831001 learned_addr learned my addr 172.21.15.1:0/1310831001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f860bfff640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 0x7f86140b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86140cd830 0x7f861415d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 -- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 msgr2=0x7f86140b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 0x7f86140b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 -- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86140cd830 msgr2=0x7f861415d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86140cd830 0x7f861415d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 -- 172.21.15.1:0/1310831001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86040094d0 con 0x7f86140d00e0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f860bfff640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 0x7f86140b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f861359e640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f860c00efc0 tx=0x7f860c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c014480 con 0x7f86140d00e0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860c00f040 con 0x7f86140d00e0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861415df00 con 0x7f86140d00e0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c014820 con 0x7f86140d00e0 2024-03-24T10:36:35.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.132+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861415e4c0 con 0x7f86140d00e0 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.133+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8614155900 con 0x7f86140d00e0 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.134+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f860c014980 con 0x7f86140d00e0 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.134+0000 7f860b7fe640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 0x7f85f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.134+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f860c08ac90 con 0x7f86140d00e0 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.134+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 0x7f85f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.135+0000 7f8612d9d640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 0x7f85f406dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8604036000 tx=0x7f860403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.139+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8614155900 con 0x7f86140d00e0 2024-03-24T10:36:35.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.322+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f86140cd310 con 0x7f85f406b810 2024-03-24T10:36:35.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.322+0000 7f860b7fe640 1 -- 172.21.15.1:0/1310831001 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8604031c40 con 0x7f85f406b810 2024-03-24T10:36:35.323 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-03-24T10:36:35.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 msgr2=0x7f85f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 0x7f85f406dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8604036000 tx=0x7f860403c040 comp rx=0 tx=0).stop 2024-03-24T10:36:35.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 msgr2=0x7f86140b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f860c00efc0 tx=0x7f860c00c490 comp rx=0 tx=0).stop 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 shutdown_connections 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f85f406b810 0x7f85f406dcd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f86140d00e0 0x7f86140b4090 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f86140cfd00 0x7f86140b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 --2- 172.21.15.1:0/1310831001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86140cd830 0x7f861415d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 >> 172.21.15.1:0/1310831001 conn(0x7f86140d1c80 msgr2=0x7f86140b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 shutdown_connections 2024-03-24T10:36:35.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.325+0000 7f8619015640 1 -- 172.21.15.1:0/1310831001 wait complete. 2024-03-24T10:36:35.336 INFO:tasks.ceph:Scrubbing osd.2 2024-03-24T10:36:35.336 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:35.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.440+0000 7fcff8a2c640 1 Processor -- start 2024-03-24T10:36:35.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 -- start start 2024-03-24T10:36:35.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcff415e600 0x7fcff4162600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff40b4fe0 0x7fcff40b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff415eb50 con 0x7fcff40af3f0 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcff4162b40 con 0x7fcff415e600 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.441+0000 7fcff8a2c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcff40b4a20 con 0x7fcff40b4fe0 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcff415e600 0x7fcff4162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcff415e600 0x7fcff4162600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35326/0 (socket says 172.21.15.1:35326) 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 -- 172.21.15.1:0/1458444555 learned_addr learned my addr 172.21.15.1:0/1458444555 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcff40af3f0 0x7fcff40af7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42736/0 (socket says 172.21.15.1:42736) 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff2d9d640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff40b4fe0 0x7fcff40b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 -- 172.21.15.1:0/1458444555 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff40b4fe0 msgr2=0x7fcff40b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:35.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff40b4fe0 0x7fcff40b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 -- 172.21.15.1:0/1458444555 >> v1:172.21.15.1:6789/0 conn(0x7fcff40af3f0 legacy=0x7fcff40af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff1d9b640 1 -- 172.21.15.1:0/1458444555 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff415ee30 con 0x7fcff415e600 2024-03-24T10:36:35.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.442+0000 7fcff159a640 1 -- 172.21.15.1:0/1458444555 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552966353 0 0) 0x7fcff415eb50 con 0x7fcff40af3f0 2024-03-24T10:36:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff415e600 0x7fcff4162600 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009870 tx=0x7fcfe802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf8d0a788b3bdea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff159a640 1 -- 172.21.15.1:0/1458444555 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8031640 con 0x7fcff415e600 2024-03-24T10:36:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff159a640 1 -- 172.21.15.1:0/1458444555 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe80317a0 con 0x7fcff415e600 2024-03-24T10:36:35.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff159a640 1 -- 172.21.15.1:0/1458444555 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8031b30 con 0x7fcff415e600 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 -- 172.21.15.1:0/1458444555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff415e600 msgr2=0x7fcff4162600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff415e600 0x7fcff4162600 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009870 tx=0x7fcfe802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 -- 172.21.15.1:0/1458444555 shutdown_connections 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff40b4fe0 0x7fcff40b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/1458444555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff415e600 0x7fcff4162600 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.443+0000 7fcff8a2c640 1 -- 172.21.15.1:0/1458444555 >> 172.21.15.1:0/1458444555 conn(0x7fcff4072410 msgr2=0x7fcff406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.444+0000 7fcff8a2c640 1 -- 172.21.15.1:0/1458444555 shutdown_connections 2024-03-24T10:36:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.444+0000 7fcff8a2c640 1 -- 172.21.15.1:0/1458444555 wait complete. 2024-03-24T10:36:35.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.445+0000 7fcff8a2c640 1 Processor -- start 2024-03-24T10:36:35.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.445+0000 7fcff8a2c640 1 -- start start 2024-03-24T10:36:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 0x7fcff4158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcff40b4fe0 0x7fcff41413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff415e600 0x7fcff41f7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcff415a600 con 0x7fcff40af3f0 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fcff415a810 con 0x7fcff40b4fe0 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fcff4141910 con 0x7fcff415e600 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 0x7fcff4158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 0x7fcff4158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35338/0 (socket says 172.21.15.1:35338) 2024-03-24T10:36:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 -- 172.21.15.1:0/3425514749 learned_addr learned my addr 172.21.15.1:0/3425514749 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcff40b4fe0 0x7fcff41413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff2d9d640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff415e600 0x7fcff41f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 -- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff415e600 msgr2=0x7fcff41f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff415e600 0x7fcff41f7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 -- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcff40b4fe0 msgr2=0x7fcff41413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcff40b4fe0 0x7fcff41413d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe80094d0 con 0x7fcff40af3f0 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff2d9d640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fcff415e600 0x7fcff41f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:35.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcff40b4fe0 0x7fcff41413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff259c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 0x7fcff4158d80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c430 tx=0x7fcfec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0074e0 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec007660 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff415ab10 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec004a20 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.446+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff415b350 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.447+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcff40af0d0 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcfec004b80 con 0x7fcff40af3f0 2024-03-24T10:36:35.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcfd406b810 0x7fcfd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fcfec089cb0 con 0x7fcff40af3f0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcfd406b810 0x7fcfd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7fcfd40716d0 0x7fcfd4073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfd40741a0 con 0x7fcfd40716d0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.448+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fcfec08c370 con 0x7fcff40af3f0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.449+0000 7fcff2d9d640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7fcfd40716d0 0x7fcfd4073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.449+0000 7fcff1d9b640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcfd406b810 0x7fcfd406dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8036000 tx=0x7fcfe803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.449+0000 7fcff2d9d640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7fcfd40716d0 0x7fcfd4073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.449+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== osd.2 v2:172.21.15.1:6816/1358018271 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd40741a0 con 0x7fcfd40716d0 2024-03-24T10:36:35.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 --> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcff415f3f0 con 0x7fcfd40716d0 2024-03-24T10:36:35.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcfe2ffd640 1 -- 172.21.15.1:0/3425514749 <== osd.2 v2:172.21.15.1:6816/1358018271 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcfe4009c30 con 0x7fcfd40716d0 2024-03-24T10:36:35.463 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:35.464 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:35.464 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7fcfd40716d0 msgr2=0x7fcfd4073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6816/1358018271,v1:172.21.15.1:6818/1358018271] conn(0x7fcfd40716d0 0x7fcfd4073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcfd406b810 msgr2=0x7fcfd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fcfd406b810 0x7fcfd406dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8036000 tx=0x7fcfe803c040 comp rx=0 tx=0).stop 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 msgr2=0x7fcff4158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 --2- 172.21.15.1:0/3425514749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff40af3f0 0x7fcff4158d80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c430 tx=0x7fcfec00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff2d9d640 1 -- 172.21.15.1:0/3425514749 reap_dead start 2024-03-24T10:36:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 shutdown_connections 2024-03-24T10:36:35.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 >> 172.21.15.1:0/3425514749 conn(0x7fcff4072410 msgr2=0x7fcff406c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 shutdown_connections 2024-03-24T10:36:35.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.463+0000 7fcff8a2c640 1 -- 172.21.15.1:0/3425514749 wait complete. 2024-03-24T10:36:35.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-24T10:36:35.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.576+0000 7f016953e640 1 Processor -- start 2024-03-24T10:36:35.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.576+0000 7f016953e640 1 -- start start 2024-03-24T10:36:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f016953e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016406d2d0 0x7f016415d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f016953e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01640d10e0 con 0x7f016406c770 2024-03-24T10:36:35.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f016953e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01640b4c70 con 0x7f01640b74c0 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f016953e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f01640d1340 con 0x7f016406d2d0 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016406d2d0 0x7f016415d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f016406d2d0 0x7f016415d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35350/0 (socket says 172.21.15.1:35350) 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 -- 172.21.15.1:0/444096760 learned_addr learned my addr 172.21.15.1:0/444096760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01627fc640 1 --1- 172.21.15.1:0/444096760 >> v1:172.21.15.1:6789/0 conn(0x7f016406c770 0x7f016406cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42746/0 (socket says 172.21.15.1:42746) 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 -- 172.21.15.1:0/444096760 >> v1:172.21.15.3:6789/0 conn(0x7f01640b74c0 legacy=0x7f01640d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 -- 172.21.15.1:0/444096760 >> v1:172.21.15.1:6789/0 conn(0x7f016406c770 legacy=0x7f016406cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.577+0000 7f01637fe640 1 -- 172.21.15.1:0/444096760 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01640b4f50 con 0x7f016406d2d0 2024-03-24T10:36:35.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.578+0000 7f01637fe640 1 --2- 172.21.15.1:0/444096760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016406d2d0 0x7f016415d8f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a339cc4fc258311 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.578+0000 7f0161ffb640 1 -- 172.21.15.1:0/444096760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c030860 con 0x7f016406d2d0 2024-03-24T10:36:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.578+0000 7f0161ffb640 1 -- 172.21.15.1:0/444096760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015c0309c0 con 0x7f016406d2d0 2024-03-24T10:36:35.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.578+0000 7f0161ffb640 1 -- 172.21.15.1:0/444096760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c030d50 con 0x7f016406d2d0 2024-03-24T10:36:35.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 -- 172.21.15.1:0/444096760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016406d2d0 msgr2=0x7f016415d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 --2- 172.21.15.1:0/444096760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016406d2d0 0x7f016415d8f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 -- 172.21.15.1:0/444096760 shutdown_connections 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 --2- 172.21.15.1:0/444096760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f016406d2d0 0x7f016415d8f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 -- 172.21.15.1:0/444096760 >> 172.21.15.1:0/444096760 conn(0x7f01640d1c80 msgr2=0x7f01640d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 -- 172.21.15.1:0/444096760 shutdown_connections 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.579+0000 7f016953e640 1 -- 172.21.15.1:0/444096760 wait complete. 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.580+0000 7f016953e640 1 Processor -- start 2024-03-24T10:36:35.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.580+0000 7f016953e640 1 -- start start 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016406c770 0x7f01641e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016406d2d0 0x7f01641e7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f01641e7750 con 0x7f01640b74c0 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f016415b760 con 0x7f016406d2d0 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f016953e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f01641f3bb0 con 0x7f016406c770 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35360/0 (socket says 172.21.15.1:35360) 2024-03-24T10:36:35.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 -- 172.21.15.1:0/3441774530 learned_addr learned my addr 172.21.15.1:0/3441774530 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01627fc640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016406d2d0 0x7f01641e7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f0162ffd640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016406c770 0x7f01641e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 -- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016406c770 msgr2=0x7f01641e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016406c770 0x7f01641e6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 -- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016406d2d0 msgr2=0x7f01641e7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016406d2d0 0x7f01641e7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 -- 172.21.15.1:0/3441774530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015c0094d0 con 0x7f01640b74c0 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.581+0000 7f01637fe640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.582+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c043400 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.582+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015c043580 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.582+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c043930 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.582+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016415ae30 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.582+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016415b3f0 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.583+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016415b910 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.583+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f015c043a90 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.583+0000 7f015b7fe640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 0x7f014406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.583+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f015c03e070 con 0x7f01640b74c0 2024-03-24T10:36:35.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.583+0000 7f0162ffd640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 0x7f014406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.584+0000 7f0162ffd640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 0x7f014406dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0154005520 tx=0x7f0154009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.588+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f015c0e0e10 con 0x7f01640b74c0 2024-03-24T10:36:35.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.766+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f01640d2a60 con 0x7f014406b810 2024-03-24T10:36:35.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.767+0000 7f015b7fe640 1 -- 172.21.15.1:0/3441774530 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f01640d2a60 con 0x7f014406b810 2024-03-24T10:36:35.769 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-03-24T10:36:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 msgr2=0x7f014406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 0x7f014406dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0154005520 tx=0x7f0154009040 comp rx=0 tx=0).stop 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 msgr2=0x7f016415f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f015c009870 tx=0x7f015c004730 comp rx=0 tx=0).stop 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 shutdown_connections 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f014406b810 0x7f014406dcd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01640b74c0 0x7f016415f200 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f016406d2d0 0x7f01641e7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 --2- 172.21.15.1:0/3441774530 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f016406c770 0x7f01641e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 >> 172.21.15.1:0/3441774530 conn(0x7f01640d1c80 msgr2=0x7f0164070690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 shutdown_connections 2024-03-24T10:36:35.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.769+0000 7f016953e640 1 -- 172.21.15.1:0/3441774530 wait complete. 2024-03-24T10:36:35.780 INFO:tasks.ceph:Scrubbing osd.3 2024-03-24T10:36:35.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.885+0000 7fd8e5d4b640 1 Processor -- start 2024-03-24T10:36:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.885+0000 7fd8e5d4b640 1 -- start start 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8e5d4b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd8e00b31d0 0x7fd8e00d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8e5d4b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8e5d4b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e00cc250 con 0x7fd8e00b1380 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8e5d4b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e00b35b0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8e5d4b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e00cc4a0 con 0x7fd8e00d0e90 2024-03-24T10:36:35.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8deffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd8e00b31d0 0x7fd8e00d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd8e00b31d0 0x7fd8e00d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:53782/0 (socket says 172.21.15.1:53782) 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3014622216 learned_addr learned my addr 172.21.15.1:0/3014622216 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8deffd640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47482/0 (socket says 172.21.15.1:47482) 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 msgr2=0x7fd8e006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.886+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3014622216 >> v1:172.21.15.3:6789/0 conn(0x7fd8e00b1380 legacy=0x7fd8e00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.887+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3014622216 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e00cc780 con 0x7fd8e00b31d0 2024-03-24T10:36:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.887+0000 7fd8deffd640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.887+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00d0950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009870 tx=0x7fd8d002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fca7636f3048a207 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.887+0000 7fd8de7fc640 1 -- 172.21.15.1:0/3014622216 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0030840 con 0x7fd8e00b31d0 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.887+0000 7fd8de7fc640 1 -- 172.21.15.1:0/3014622216 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d00309a0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8de7fc640 1 -- 172.21.15.1:0/3014622216 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0030d30 con 0x7fd8e00b31d0 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00b31d0 msgr2=0x7fd8e00d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00d0950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009870 tx=0x7fd8d002b980 comp rx=0 tx=0).stop 2024-03-24T10:36:35.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/3014622216 shutdown_connections 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00d0e90 0x7fd8e006cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/3014622216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00d0950 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/3014622216 >> 172.21.15.1:0/3014622216 conn(0x7fd8e00d1c00 msgr2=0x7fd8e00d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/3014622216 shutdown_connections 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.888+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/3014622216 wait complete. 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.889+0000 7fd8e5d4b640 1 Processor -- start 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.889+0000 7fd8e5d4b640 1 -- start start 2024-03-24T10:36:35.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00b1380 0x7fd8e00c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00d0e90 0x7fd8e00cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e00c7460 con 0x7fd8e00b31d0 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0155360 con 0x7fd8e00d0e90 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8e5d4b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e0154820 con 0x7fd8e00b1380 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35372/0 (socket says 172.21.15.1:35372) 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 -- 172.21.15.1:0/2804129201 learned_addr learned my addr 172.21.15.1:0/2804129201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:35.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8dffff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00d0e90 0x7fd8e00cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00b1380 0x7fd8e00c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 -- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00b1380 msgr2=0x7fd8e00c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00b1380 0x7fd8e00c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 -- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00d0e90 msgr2=0x7fd8e00cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.890+0000 7fd8deffd640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00d0e90 0x7fd8e00cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8deffd640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d4009400 con 0x7fd8e00b31d0 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8dffff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8e00d0e90 0x7fd8e00cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:35.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd8e00b1380 0x7fd8e00c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8deffd640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00ca630 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400e960 tx=0x7fd8d400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400ca00 con 0x7fd8e00b31d0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d400cb60 con 0x7fd8e00b31d0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d00094d0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4013470 con 0x7fd8e00b31d0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.891+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e00c7180 con 0x7fd8e00b31d0 2024-03-24T10:36:35.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.892+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd8e00ccd40 con 0x7fd8e00b31d0 2024-03-24T10:36:35.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.892+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd8d40135d0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8c7fff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd8cc06b810 0x7fd8cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fd8d408a0f0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd8cc06b810 0x7fd8cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8c7fff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7fd8cc0716d0 0x7fd8cc073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8cc0741a0 con 0x7fd8cc0716d0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd8d408c7b0 con 0x7fd8e00b31d0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8dffff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7fd8cc0716d0 0x7fd8cc073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd8cc06b810 0x7fd8cc06dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0036000 tx=0x7fd8d003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.893+0000 7fd8dffff640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7fd8cc0716d0 0x7fd8cc073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:35.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.894+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== osd.3 v2:172.21.15.1:6817/2415676065 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd8cc0741a0 con 0x7fd8cc0716d0 2024-03-24T10:36:35.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.907+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 --> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd8e004f470 con 0x7fd8cc0716d0 2024-03-24T10:36:35.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.907+0000 7fd8c7fff640 1 -- 172.21.15.1:0/2804129201 <== osd.3 v2:172.21.15.1:6817/2415676065 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd8e004f470 con 0x7fd8cc0716d0 2024-03-24T10:36:35.908 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:35.908 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:35.908 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:35.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7fd8cc0716d0 msgr2=0x7fd8cc073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6817/2415676065,v1:172.21.15.1:6819/2415676065] conn(0x7fd8cc0716d0 0x7fd8cc073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:35.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd8cc06b810 msgr2=0x7fd8cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd8cc06b810 0x7fd8cc06dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0036000 tx=0x7fd8d003b040 comp rx=0 tx=0).stop 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 msgr2=0x7fd8e00ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 --2- 172.21.15.1:0/2804129201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00b31d0 0x7fd8e00ca630 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400e960 tx=0x7fd8d400ee20 comp rx=0 tx=0).stop 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8dffff640 1 -- 172.21.15.1:0/2804129201 reap_dead start 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 shutdown_connections 2024-03-24T10:36:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 >> 172.21.15.1:0/2804129201 conn(0x7fd8e00d1c00 msgr2=0x7fd8e006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:35.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 shutdown_connections 2024-03-24T10:36:35.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:35.908+0000 7fd8e5d4b640 1 -- 172.21.15.1:0/2804129201 wait complete. 2024-03-24T10:36:35.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-24T10:36:36.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.021+0000 7fda8638a640 1 Processor -- start 2024-03-24T10:36:36.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.021+0000 7fda8638a640 1 -- start start 2024-03-24T10:36:36.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda8638a640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fda800b74c0 0x7fda800d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda8638a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda8006d2d0 0x7fda8015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda8638a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda800b4c70 con 0x7fda8006c770 2024-03-24T10:36:36.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda8638a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fda800d1340 con 0x7fda8006d2d0 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda8638a640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fda800b4f50 con 0x7fda800b74c0 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda8006d2d0 0x7fda8015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda8006d2d0 0x7fda8015d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35378/0 (socket says 172.21.15.1:35378) 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 -- 172.21.15.1:0/509144652 learned_addr learned my addr 172.21.15.1:0/509144652 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda7ffff640 1 --2- 172.21.15.1:0/509144652 >> v2:172.21.15.3:3300/0 conn(0x7fda800b74c0 0x7fda800d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 -- 172.21.15.1:0/509144652 >> v2:172.21.15.3:3300/0 conn(0x7fda800b74c0 msgr2=0x7fda800d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 --2- 172.21.15.1:0/509144652 >> v2:172.21.15.3:3300/0 conn(0x7fda800b74c0 0x7fda800d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 -- 172.21.15.1:0/509144652 >> v1:172.21.15.3:6789/0 conn(0x7fda8006c770 legacy=0x7fda8006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.022+0000 7fda84927640 1 -- 172.21.15.1:0/509144652 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda800b5230 con 0x7fda8006d2d0 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.023+0000 7fda84927640 1 --2- 172.21.15.1:0/509144652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8015d8f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fda8006bba0 tx=0x7fda7802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e16743df021d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.023+0000 7fda7effd640 1 -- 172.21.15.1:0/509144652 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7802ca80 con 0x7fda8006d2d0 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.023+0000 7fda7effd640 1 -- 172.21.15.1:0/509144652 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7802cbe0 con 0x7fda8006d2d0 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.023+0000 7fda7effd640 1 -- 172.21.15.1:0/509144652 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780343f0 con 0x7fda8006d2d0 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 -- 172.21.15.1:0/509144652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 msgr2=0x7fda8015d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 --2- 172.21.15.1:0/509144652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8015d8f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fda8006bba0 tx=0x7fda7802bee0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 -- 172.21.15.1:0/509144652 shutdown_connections 2024-03-24T10:36:36.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 --2- 172.21.15.1:0/509144652 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8015d8f0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 --2- 172.21.15.1:0/509144652 >> v2:172.21.15.3:3300/0 conn(0x7fda800b74c0 0x7fda800d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 -- 172.21.15.1:0/509144652 >> 172.21.15.1:0/509144652 conn(0x7fda800d1c80 msgr2=0x7fda800d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 -- 172.21.15.1:0/509144652 shutdown_connections 2024-03-24T10:36:36.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.024+0000 7fda8638a640 1 -- 172.21.15.1:0/509144652 wait complete. 2024-03-24T10:36:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.025+0000 7fda8638a640 1 Processor -- start 2024-03-24T10:36:36.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.025+0000 7fda8638a640 1 -- start start 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8006c770 0x7fda801e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda800b74c0 0x7fda801e6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fda801e88c0 con 0x7fda8006d2d0 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fda801e72e0 con 0x7fda8006c770 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda8638a640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fda801e75c0 con 0x7fda800b74c0 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35392/0 (socket says 172.21.15.1:35392) 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 -- 172.21.15.1:0/2758885428 learned_addr learned my addr 172.21.15.1:0/2758885428 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 -- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda800b74c0 msgr2=0x7fda801e6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda800b74c0 0x7fda801e6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 -- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8006c770 msgr2=0x7fda801e8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7ffff640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8006c770 0x7fda801e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8006c770 0x7fda801e8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 -- 172.21.15.1:0/2758885428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda78009b30 con 0x7fda8006d2d0 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.026+0000 7fda7f7fe640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c410 tx=0x7fda7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda700035f0 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda70003750 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70003b40 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda801e78a0 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda801e7e60 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.027+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda801e8050 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.028+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda70003ca0 con 0x7fda8006d2d0 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.028+0000 7fda7cff9640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 0x7fda6006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.028+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fda700895b0 con 0x7fda8006d2d0 2024-03-24T10:36:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.028+0000 7fda7ffff640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 0x7fda6006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.029+0000 7fda7ffff640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 0x7fda6006dd00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fda6800b7c0 tx=0x7fda68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.033+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda700542c0 con 0x7fda8006d2d0 2024-03-24T10:36:36.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.214+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fda800d0f10 con 0x7fda6006b840 2024-03-24T10:36:36.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.214+0000 7fda7cff9640 1 -- 172.21.15.1:0/2758885428 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda6800c040 con 0x7fda6006b840 2024-03-24T10:36:36.215 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-03-24T10:36:36.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.216+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 msgr2=0x7fda6006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 0x7fda6006dd00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fda6800b7c0 tx=0x7fda68008040 comp rx=0 tx=0).stop 2024-03-24T10:36:36.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 msgr2=0x7fda8014e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c410 tx=0x7fda7000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 shutdown_connections 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda6006b840 0x7fda6006dd00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda800b74c0 0x7fda801e6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda8006d2d0 0x7fda8014e070 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 --2- 172.21.15.1:0/2758885428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8006c770 0x7fda801e8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 >> 172.21.15.1:0/2758885428 conn(0x7fda800d1c80 msgr2=0x7fda800d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 shutdown_connections 2024-03-24T10:36:36.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.217+0000 7fda8638a640 1 -- 172.21.15.1:0/2758885428 wait complete. 2024-03-24T10:36:36.227 INFO:tasks.ceph:Scrubbing osd.4 2024-03-24T10:36:36.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:36.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.333+0000 7fa4ea615640 1 Processor -- start 2024-03-24T10:36:36.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.333+0000 7fa4ea615640 1 -- start start 2024-03-24T10:36:36.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4ea615640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4e406eb90 0x7fa4e40b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4ea615640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e40b3a80 con 0x7fa4e40b3f10 2024-03-24T10:36:36.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4ea615640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e40cfa70 con 0x7fa4e415ac00 2024-03-24T10:36:36.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4ea615640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fa4e40d0030 con 0x7fa4e406eb90 2024-03-24T10:36:36.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e8bb2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa4e40b3f10 0x7fa4e40b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42748/0 (socket says 172.21.15.1:42748) 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2049275868 learned_addr learned my addr 172.21.15.1:0/2049275868 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e37fe640 1 --2- 172.21.15.1:0/2049275868 >> v2:172.21.15.3:3300/0 conn(0x7fa4e406eb90 0x7fa4e40b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869548557 0 0) 0x7fa4e40b3a80 con 0x7fa4e40b3f10 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4cc003520 con 0x7fa4e40b3f10 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3566921975 0 0) 0x7fa4cc003520 con 0x7fa4e40b3f10 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 >> v2:172.21.15.3:3300/0 conn(0x7fa4e406eb90 msgr2=0x7fa4e40b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 --2- 172.21.15.1:0/2049275868 >> v2:172.21.15.3:3300/0 conn(0x7fa4e406eb90 0x7fa4e40b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.334+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 >> v1:172.21.15.3:6789/0 conn(0x7fa4e415ac00 legacy=0x7fa4e406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e40cfd50 con 0x7fa4e40b3f10 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fa4dc0019f0 con 0x7fa4e40b3f10 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 >> v1:172.21.15.1:6789/0 conn(0x7fa4e40b3f10 legacy=0x7fa4e40b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4cc003970 0x7fa4cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 0x7fa4cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e40d0030 con 0x7fa4cc005180 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa4cc00c020 con 0x7fa4cc0047c0 2024-03-24T10:36:36.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa4cc00c1e0 con 0x7fa4cc003970 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 0x7fa4cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e37fe640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e3fff640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4cc003970 0x7fa4cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4cc003970 msgr2=0x7fa4cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4cc003970 0x7fa4cc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc0047c0 msgr2=0x7fa4cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.335+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2049275868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4cc003520 con 0x7fa4cc005180 2024-03-24T10:36:36.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2049275868 reap_dead start 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 0x7fa4cc00b980 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00a830 tx=0x7fa4cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef940453ff3fc44 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0072d0 con 0x7fa4cc005180 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc02f070 con 0x7fa4cc005180 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4e2ffd640 1 -- 172.21.15.1:0/2049275868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc007650 con 0x7fa4cc005180 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4ea615640 1 -- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 msgr2=0x7fa4cc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4ea615640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 0x7fa4cc00b980 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00a830 tx=0x7fa4cc0057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4ea615640 1 -- 172.21.15.1:0/2049275868 shutdown_connections 2024-03-24T10:36:36.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4ea615640 1 --2- 172.21.15.1:0/2049275868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4cc005180 0x7fa4cc00b980 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.336+0000 7fa4ea615640 1 -- 172.21.15.1:0/2049275868 >> 172.21.15.1:0/2049275868 conn(0x7fa4e4072410 msgr2=0x7fa4e406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.337+0000 7fa4ea615640 1 -- 172.21.15.1:0/2049275868 shutdown_connections 2024-03-24T10:36:36.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.337+0000 7fa4ea615640 1 -- 172.21.15.1:0/2049275868 wait complete. 2024-03-24T10:36:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.338+0000 7fa4ea615640 1 Processor -- start 2024-03-24T10:36:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.338+0000 7fa4ea615640 1 -- start start 2024-03-24T10:36:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc005180 0x7fa4e41404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 0x7fa4e4142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4e4143430 0x7fa4e4152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e4153340 con 0x7fa4e4144350 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41419a0 con 0x7fa4cc005180 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e41438c0 con 0x7fa4e4143430 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 0x7fa4e4142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 0x7fa4e4142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35416/0 (socket says 172.21.15.1:35416) 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 -- 172.21.15.1:0/2089532619 learned_addr learned my addr 172.21.15.1:0/2089532619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e3fff640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc005180 0x7fa4e41404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4e4143430 0x7fa4e4152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 -- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4e4143430 msgr2=0x7fa4e4152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fa4e4143430 0x7fa4e4152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 -- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc005180 msgr2=0x7fa4e41404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc005180 0x7fa4e41404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e4141000 con 0x7fa4e4144350 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e3fff640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4cc005180 0x7fa4e41404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e37fe640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 0x7fa4e4142ef0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40118e0 tx=0x7fa4d4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d400faf0 con 0x7fa4e4144350 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d400fc50 con 0x7fa4e4144350 2024-03-24T10:36:36.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4014440 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e41412e0 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.339+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e4140a00 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.340+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa4e415afe0 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa4d40145a0 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4d006b810 0x7fa4d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fa4d408a770 con 0x7fa4e4144350 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e3fff640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4d006b810 0x7fa4d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7fa4d00716d0 0x7fa4d0073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4d00741a0 con 0x7fa4d00716d0 2024-03-24T10:36:36.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.341+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fa4d408ce30 con 0x7fa4e4144350 2024-03-24T10:36:36.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.342+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7fa4d00716d0 0x7fa4d0073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.342+0000 7fa4e3fff640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4d006b810 0x7fa4d006dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005580 tx=0x7fa4d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.342+0000 7fa4e8bb2640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7fa4d00716d0 0x7fa4d0073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.343+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== osd.4 v2:172.21.15.3:6800/3439594233 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa4d00741a0 con 0x7fa4d00716d0 2024-03-24T10:36:36.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.356+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 --> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa4e4141c10 con 0x7fa4d00716d0 2024-03-24T10:36:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4e0ff9640 1 -- 172.21.15.1:0/2089532619 <== osd.4 v2:172.21.15.3:6800/3439594233 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa4e4141c10 con 0x7fa4d00716d0 2024-03-24T10:36:36.358 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:36.358 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:36.358 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7fa4d00716d0 msgr2=0x7fa4d0073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.3:6800/3439594233,v1:172.21.15.3:6801/3439594233] conn(0x7fa4d00716d0 0x7fa4d0073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4d006b810 msgr2=0x7fa4d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fa4d006b810 0x7fa4d006dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005580 tx=0x7fa4d8009040 comp rx=0 tx=0).stop 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 msgr2=0x7fa4e4142ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 --2- 172.21.15.1:0/2089532619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4e4144350 0x7fa4e4142ef0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40118e0 tx=0x7fa4d4011da0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4e8bb2640 1 -- 172.21.15.1:0/2089532619 reap_dead start 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 shutdown_connections 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 >> 172.21.15.1:0/2089532619 conn(0x7fa4e4072410 msgr2=0x7fa4e415daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 shutdown_connections 2024-03-24T10:36:36.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.357+0000 7fa4ea615640 1 -- 172.21.15.1:0/2089532619 wait complete. 2024-03-24T10:36:36.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-24T10:36:36.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.470+0000 7fda5d6d4640 1 Processor -- start 2024-03-24T10:36:36.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.470+0000 7fda5d6d4640 1 -- start start 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda5d6d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda580d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda5d6d4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda580cdd60 con 0x7fda58070b90 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda5d6d4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5806e200 con 0x7fda580b2aa0 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda5d6d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fda5806eaa0 con 0x7fda58144770 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda4ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda58070b90 0x7fda5806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42762/0 (socket says 172.21.15.1:42762) 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda4ffff640 1 -- 172.21.15.1:0/1945180148 learned_addr learned my addr 172.21.15.1:0/1945180148 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317279542 0 0) 0x7fda580cdd60 con 0x7fda58070b90 2024-03-24T10:36:36.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda56ffd640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda580d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda44003520 con 0x7fda58070b90 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3152776024 0 0) 0x7fda44003520 con 0x7fda58070b90 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 msgr2=0x7fda580d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda580d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 >> v1:172.21.15.3:6789/0 conn(0x7fda580b2aa0 legacy=0x7fda580ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5806e4e0 con 0x7fda58070b90 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.471+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2661282717 0 0) 0x7fda40001670 con 0x7fda58070b90 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 >> v1:172.21.15.1:6789/0 conn(0x7fda58070b90 legacy=0x7fda5806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda440039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44003f20 0x7fda440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda44004d00 0x7fda44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fda5806eaa0 con 0x7fda44004d00 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fda4400ba60 con 0x7fda44003f20 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fda4400bc20 con 0x7fda58144770 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda44004d00 0x7fda44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda4ffff640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44003f20 0x7fda440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda56ffd640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda440039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 -- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 msgr2=0x7fda440039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda58144770 0x7fda440039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 -- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44003f20 msgr2=0x7fda440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44003f20 0x7fda440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.472+0000 7fda577fe640 1 -- 172.21.15.1:0/1945180148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda44003520 con 0x7fda44004d00 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda577fe640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda44004d00 0x7fda44005180 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda50001e50 tx=0x7fda440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e12a6ac1cd6830 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50003b70 con 0x7fda44004d00 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50003cd0 con 0x7fda44004d00 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda567fc640 1 -- 172.21.15.1:0/1945180148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50005310 con 0x7fda44004d00 2024-03-24T10:36:36.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1945180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda44004d00 msgr2=0x7fda44005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1945180148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda44004d00 0x7fda44005180 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda50001e50 tx=0x7fda440057f0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda577fe640 1 -- 172.21.15.1:0/1945180148 reap_dead start 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1945180148 shutdown_connections 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1945180148 >> 172.21.15.1:0/1945180148 conn(0x7fda580b17a0 msgr2=0x7fda580b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1945180148 shutdown_connections 2024-03-24T10:36:36.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.473+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1945180148 wait complete. 2024-03-24T10:36:36.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 Processor -- start 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 -- start start 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44004d00 0x7fda581f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda580ae070 0x7fda581426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fda580adbe0 con 0x7fda581f7410 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fda581f4d90 con 0x7fda44004d00 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda5d6d4640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fda580ad8e0 con 0x7fda580ae070 2024-03-24T10:36:36.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda4ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda4ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35422/0 (socket says 172.21.15.1:35422) 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.475+0000 7fda4ffff640 1 -- 172.21.15.1:0/1886452517 learned_addr learned my addr 172.21.15.1:0/1886452517 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda56ffd640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44004d00 0x7fda581f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda577fe640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda580ae070 0x7fda581426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 -- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda580ae070 msgr2=0x7fda581426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda580ae070 0x7fda581426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 -- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44004d00 msgr2=0x7fda581f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44004d00 0x7fda581f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 -- 172.21.15.1:0/1886452517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda581f7c60 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4ffff640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fda40007290 tx=0x7fda40007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4000f9c0 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4000fb20 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda581f7f40 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4000fed0 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.476+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda581f7890 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.477+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5804f4e0 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.478+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda4001d460 con 0x7fda581f7410 2024-03-24T10:36:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.478+0000 7fda4f7fe640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 0x7fda3806dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.478+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fda4008a130 con 0x7fda581f7410 2024-03-24T10:36:36.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.478+0000 7fda56ffd640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 0x7fda3806dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.478+0000 7fda56ffd640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 0x7fda3806dc70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fda48009b00 tx=0x7fda48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.482+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda40054e50 con 0x7fda581f7410 2024-03-24T10:36:36.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.666+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fda5806de90 con 0x7fda3806b7b0 2024-03-24T10:36:36.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.666+0000 7fda4f7fe640 1 -- 172.21.15.1:0/1886452517 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fda5806de90 con 0x7fda3806b7b0 2024-03-24T10:36:36.667 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-03-24T10:36:36.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 msgr2=0x7fda3806dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 0x7fda3806dc70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fda48009b00 tx=0x7fda48008040 comp rx=0 tx=0).stop 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 msgr2=0x7fda5815d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fda40007290 tx=0x7fda40007750 comp rx=0 tx=0).stop 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 shutdown_connections 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fda3806b7b0 0x7fda3806dc70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fda580ae070 0x7fda581426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda581f7410 0x7fda5815d050 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 --2- 172.21.15.1:0/1886452517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda44004d00 0x7fda581f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 >> 172.21.15.1:0/1886452517 conn(0x7fda580b17a0 msgr2=0x7fda580cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 shutdown_connections 2024-03-24T10:36:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.668+0000 7fda5d6d4640 1 -- 172.21.15.1:0/1886452517 wait complete. 2024-03-24T10:36:36.679 INFO:tasks.ceph:Scrubbing osd.5 2024-03-24T10:36:36.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.784+0000 7faf88925640 1 Processor -- start 2024-03-24T10:36:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.784+0000 7faf88925640 1 -- start start 2024-03-24T10:36:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf88925640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 0x7faf8415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf88925640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf88925640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf8415f580 con 0x7faf8415ead0 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf88925640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7faf8406e760 con 0x7faf840cffc0 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf88925640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faf8415efc0 con 0x7faf8406ddd0 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf8259c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 0x7faf8415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47566/0 (socket says 172.21.15.1:47566) 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf8259c640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 0x7faf8415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:53850/0 (socket says 172.21.15.1:53850) 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 -- 172.21.15.1:0/3316896321 learned_addr learned my addr 172.21.15.1:0/3316896321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf81d9b640 1 --1- 172.21.15.1:0/3316896321 >> v1:172.21.15.3:6789/0 conn(0x7faf8415ead0 0x7faf840d11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:43744/0 (socket says 172.21.15.1:43744) 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 -- 172.21.15.1:0/3316896321 >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 msgr2=0x7faf8415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 --2- 172.21.15.1:0/3316896321 >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 0x7faf8415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 -- 172.21.15.1:0/3316896321 >> v1:172.21.15.3:6789/0 conn(0x7faf8415ead0 legacy=0x7faf840d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.785+0000 7faf82d9d640 1 -- 172.21.15.1:0/3316896321 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8415f2a0 con 0x7faf8406ddd0 2024-03-24T10:36:36.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.786+0000 7faf82d9d640 1 --2- 172.21.15.1:0/3316896321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009e50 tx=0x7faf7c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e25ba3a4533df3ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.786+0000 7faf8159a640 1 -- 172.21.15.1:0/3316896321 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c02ca70 con 0x7faf8406ddd0 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.786+0000 7faf8159a640 1 -- 172.21.15.1:0/3316896321 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf7c02cbd0 con 0x7faf8406ddd0 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.786+0000 7faf8159a640 1 -- 172.21.15.1:0/3316896321 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c0343f0 con 0x7faf8406ddd0 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 -- 172.21.15.1:0/3316896321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 msgr2=0x7faf8406e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 --2- 172.21.15.1:0/3316896321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009e50 tx=0x7faf7c02bed0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 -- 172.21.15.1:0/3316896321 shutdown_connections 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 --2- 172.21.15.1:0/3316896321 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8406e220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 --2- 172.21.15.1:0/3316896321 >> v2:172.21.15.3:3300/0 conn(0x7faf840cffc0 0x7faf8415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 -- 172.21.15.1:0/3316896321 >> 172.21.15.1:0/3316896321 conn(0x7faf8406c1b0 msgr2=0x7faf8406c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 -- 172.21.15.1:0/3316896321 shutdown_connections 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.787+0000 7faf88925640 1 -- 172.21.15.1:0/3316896321 wait complete. 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.788+0000 7faf88925640 1 Processor -- start 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.788+0000 7faf88925640 1 -- start start 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8413b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 0x7faf8415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf8415ead0 0x7faf8415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faf8415ac10 con 0x7faf840cffc0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7faf8415aef0 con 0x7faf8415ead0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf88925640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7faf8415b1d0 con 0x7faf8406ddd0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 0x7faf8415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 0x7faf8415c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35430/0 (socket says 172.21.15.1:35430) 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 -- 172.21.15.1:0/2110215194 learned_addr learned my addr 172.21.15.1:0/2110215194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf82d9d640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf8415ead0 0x7faf8415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf8259c640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8413b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 -- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 msgr2=0x7faf8413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7faf8406ddd0 0x7faf8413b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 -- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf8415ead0 msgr2=0x7faf8415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf8415ead0 0x7faf8415cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.789+0000 7faf81d9b640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c009b30 con 0x7faf840cffc0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf81d9b640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 0x7faf8415c5d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7faf7800c440 tx=0x7faf7800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf780074e0 con 0x7faf840cffc0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf78007640 con 0x7faf840cffc0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8415b4b0 con 0x7faf840cffc0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78004a20 con 0x7faf840cffc0 2024-03-24T10:36:36.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.790+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8413b920 con 0x7faf840cffc0 2024-03-24T10:36:36.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.791+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf8415b060 con 0x7faf840cffc0 2024-03-24T10:36:36.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.791+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faf78004b80 con 0x7faf840cffc0 2024-03-24T10:36:36.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf72ffd640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf6406b810 0x7faf6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7faf78089740 con 0x7faf840cffc0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf8259c640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf6406b810 0x7faf6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf72ffd640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7faf640716d0 0x7faf64073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf640741a0 con 0x7faf640716d0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7faf7808be00 con 0x7faf840cffc0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf8259c640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf6406b810 0x7faf6406dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faf74009aa0 tx=0x7faf74009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.792+0000 7faf82d9d640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7faf640716d0 0x7faf64073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.793+0000 7faf82d9d640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7faf640716d0 0x7faf64073af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.793+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== osd.5 v2:172.21.15.3:6824/3670894664 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf640741a0 con 0x7faf640716d0 2024-03-24T10:36:36.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.806+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 --> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faf840d18d0 con 0x7faf640716d0 2024-03-24T10:36:36.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf72ffd640 1 -- 172.21.15.1:0/2110215194 <== osd.5 v2:172.21.15.3:6824/3670894664 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faf7c02ca60 con 0x7faf640716d0 2024-03-24T10:36:36.807 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:36.808 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:36.808 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7faf640716d0 msgr2=0x7faf64073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.3:6824/3670894664,v1:172.21.15.3:6825/3670894664] conn(0x7faf640716d0 0x7faf64073af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf6406b810 msgr2=0x7faf6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7faf6406b810 0x7faf6406dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faf74009aa0 tx=0x7faf74009190 comp rx=0 tx=0).stop 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 msgr2=0x7faf8415c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.807+0000 7faf88925640 1 --2- 172.21.15.1:0/2110215194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf840cffc0 0x7faf8415c5d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7faf7800c440 tx=0x7faf7800c900 comp rx=0 tx=0).stop 2024-03-24T10:36:36.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.808+0000 7faf82d9d640 1 -- 172.21.15.1:0/2110215194 reap_dead start 2024-03-24T10:36:36.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.808+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 shutdown_connections 2024-03-24T10:36:36.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.808+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 >> 172.21.15.1:0/2110215194 conn(0x7faf8406c1b0 msgr2=0x7faf8406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.808+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 shutdown_connections 2024-03-24T10:36:36.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.808+0000 7faf88925640 1 -- 172.21.15.1:0/2110215194 wait complete. 2024-03-24T10:36:36.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-24T10:36:36.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.919+0000 7fdc503c2640 1 Processor -- start 2024-03-24T10:36:36.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.920+0000 7fdc503c2640 1 -- start start 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.920+0000 7fdc503c2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc48154fe0 0x7fdc4806b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.920+0000 7fdc503c2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc481553c0 con 0x7fdc480ae380 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.920+0000 7fdc503c2640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc4806c7e0 con 0x7fdc4806b5e0 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.920+0000 7fdc503c2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdc4806d250 con 0x7fdc48154fe0 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e15e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc48154fe0 0x7fdc4806b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e95f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdc480ae380 0x7fdc480b3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42768/0 (socket says 172.21.15.1:42768) 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e95f640 1 -- 172.21.15.1:0/1909472087 learned_addr learned my addr 172.21.15.1:0/1909472087 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:36.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e15e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc48154fe0 0x7fdc4806b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35442/0 (socket says 172.21.15.1:35442) 2024-03-24T10:36:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e15e640 1 -- 172.21.15.1:0/1909472087 >> v1:172.21.15.3:6789/0 conn(0x7fdc4806b5e0 legacy=0x7fdc480adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e15e640 1 -- 172.21.15.1:0/1909472087 >> v1:172.21.15.1:6789/0 conn(0x7fdc480ae380 legacy=0x7fdc480b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4d15c640 1 -- 172.21.15.1:0/1909472087 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822913102 0 0) 0x7fdc481553c0 con 0x7fdc480ae380 2024-03-24T10:36:36.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.921+0000 7fdc4e15e640 1 -- 172.21.15.1:0/1909472087 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4806bbd0 con 0x7fdc48154fe0 2024-03-24T10:36:36.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc4e15e640 1 --2- 172.21.15.1:0/1909472087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc48154fe0 0x7fdc4806b0a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdc440097a0 tx=0x7fdc4402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01005847776d587 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc4d15c640 1 -- 172.21.15.1:0/1909472087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44030610 con 0x7fdc48154fe0 2024-03-24T10:36:36.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc4d15c640 1 -- 172.21.15.1:0/1909472087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc44030770 con 0x7fdc48154fe0 2024-03-24T10:36:36.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc4d15c640 1 -- 172.21.15.1:0/1909472087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44030b00 con 0x7fdc48154fe0 2024-03-24T10:36:36.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc503c2640 1 -- 172.21.15.1:0/1909472087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc48154fe0 msgr2=0x7fdc4806b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.922+0000 7fdc503c2640 1 --2- 172.21.15.1:0/1909472087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc48154fe0 0x7fdc4806b0a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdc440097a0 tx=0x7fdc4402b960 comp rx=0 tx=0).stop 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.923+0000 7fdc503c2640 1 -- 172.21.15.1:0/1909472087 shutdown_connections 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.923+0000 7fdc503c2640 1 --2- 172.21.15.1:0/1909472087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc48154fe0 0x7fdc4806b0a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.923+0000 7fdc503c2640 1 -- 172.21.15.1:0/1909472087 >> 172.21.15.1:0/1909472087 conn(0x7fdc480d0570 msgr2=0x7fdc480d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.923+0000 7fdc503c2640 1 -- 172.21.15.1:0/1909472087 shutdown_connections 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.923+0000 7fdc503c2640 1 -- 172.21.15.1:0/1909472087 wait complete. 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 Processor -- start 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 -- start start 2024-03-24T10:36:36.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc4806b5e0 0x7fdc48155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc48154fe0 0x7fdc4806e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc48155b20 con 0x7fdc480ae380 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fdc480ce610 con 0x7fdc48154fe0 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.924+0000 7fdc503c2640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdc481581d0 con 0x7fdc4806b5e0 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35454/0 (socket says 172.21.15.1:35454) 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 -- 172.21.15.1:0/18998529 learned_addr learned my addr 172.21.15.1:0/18998529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:36.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4e95f640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc48154fe0 0x7fdc4806e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4e15e640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc4806b5e0 0x7fdc48155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 -- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc4806b5e0 msgr2=0x7fdc48155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc4806b5e0 0x7fdc48155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 -- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc48154fe0 msgr2=0x7fdc4806e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc48154fe0 0x7fdc4806e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 -- 172.21.15.1:0/18998529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc44009400 con 0x7fdc480ae380 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc4d95d640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c400 tx=0x7fdc3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc380052d0 con 0x7fdc480ae380 2024-03-24T10:36:36.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc38005430 con 0x7fdc480ae380 2024-03-24T10:36:36.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc48157ef0 con 0x7fdc480ae380 2024-03-24T10:36:36.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38003830 con 0x7fdc480ae380 2024-03-24T10:36:36.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.925+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc48155740 con 0x7fdc480ae380 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.926+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4804f4e0 con 0x7fdc480ae380 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.927+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdc38003990 con 0x7fdc480ae380 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.927+0000 7fdc3effd640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.927+0000 7fdc4e15e640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.927+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fdc380893c0 con 0x7fdc480ae380 2024-03-24T10:36:36.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.928+0000 7fdc4e15e640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdc44036000 tx=0x7fdc44008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:36.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:36.931+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdc38054170 con 0x7fdc480ae380 2024-03-24T10:36:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.110+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fdc4815f5f0 con 0x7fdc2c06b810 2024-03-24T10:36:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.111+0000 7fdc3effd640 1 -- 172.21.15.1:0/18998529 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdc4815f5f0 con 0x7fdc2c06b810 2024-03-24T10:36:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-03-24T10:36:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 msgr2=0x7fdc2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdc44036000 tx=0x7fdc44008030 comp rx=0 tx=0).stop 2024-03-24T10:36:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 msgr2=0x7fdc481576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c400 tx=0x7fdc3800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 shutdown_connections 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc48154fe0 0x7fdc4806e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc480ae380 0x7fdc481576a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 --2- 172.21.15.1:0/18998529 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc4806b5e0 0x7fdc48155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 >> 172.21.15.1:0/18998529 conn(0x7fdc480d0570 msgr2=0x7fdc480aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 shutdown_connections 2024-03-24T10:36:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.113+0000 7fdc503c2640 1 -- 172.21.15.1:0/18998529 wait complete. 2024-03-24T10:36:37.123 INFO:tasks.ceph:Scrubbing osd.6 2024-03-24T10:36:37.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:37.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.230+0000 7fd56f46b640 1 Processor -- start 2024-03-24T10:36:37.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.230+0000 7fd56f46b640 1 -- start start 2024-03-24T10:36:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56f46b640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 0x7fd5680cf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56f46b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56f46b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5680b6f50 con 0x7fd5680d2540 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56f46b640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680cfeb0 con 0x7fd5680d1e20 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56f46b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5680b7230 con 0x7fd5680b7d10 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56ca06640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 0x7fd5680cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3301/0 says I am v2:172.21.15.1:47574/0 (socket says 172.21.15.1:47574) 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56d207640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd5680d2540 0x7fd5680d2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:43752/0 (socket says 172.21.15.1:43752) 2024-03-24T10:36:37.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 -- 172.21.15.1:0/2783218375 learned_addr learned my addr 172.21.15.1:0/2783218375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 -- 172.21.15.1:0/2783218375 >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 msgr2=0x7fd5680cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:37.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 --2- 172.21.15.1:0/2783218375 >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 0x7fd5680cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 -- 172.21.15.1:0/2783218375 >> v1:172.21.15.3:6789/0 conn(0x7fd5680d2540 legacy=0x7fd5680d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56da08640 1 -- 172.21.15.1:0/2783218375 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680b7510 con 0x7fd5680b7d10 2024-03-24T10:36:37.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.231+0000 7fd56ca06640 1 --2- 172.21.15.1:0/2783218375 >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 0x7fd5680cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.232+0000 7fd56da08640 1 --2- 172.21.15.1:0/2783218375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd560009e50 tx=0x7fd56002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb54702efbf6bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.232+0000 7fd55ffff640 1 -- 172.21.15.1:0/2783218375 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56002cb90 con 0x7fd5680b7d10 2024-03-24T10:36:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.232+0000 7fd55ffff640 1 -- 172.21.15.1:0/2783218375 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56002ccf0 con 0x7fd5680b7d10 2024-03-24T10:36:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.232+0000 7fd55ffff640 1 -- 172.21.15.1:0/2783218375 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560034610 con 0x7fd5680b7d10 2024-03-24T10:36:37.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 -- 172.21.15.1:0/2783218375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 msgr2=0x7fd5680b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 --2- 172.21.15.1:0/2783218375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd560009e50 tx=0x7fd56002bff0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 -- 172.21.15.1:0/2783218375 shutdown_connections 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 --2- 172.21.15.1:0/2783218375 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680b7d10 0x7fd5680b81a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 --2- 172.21.15.1:0/2783218375 >> v2:172.21.15.3:3300/0 conn(0x7fd5680d1e20 0x7fd5680cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 -- 172.21.15.1:0/2783218375 >> 172.21.15.1:0/2783218375 conn(0x7fd56809d1d0 msgr2=0x7fd5680d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 -- 172.21.15.1:0/2783218375 shutdown_connections 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.233+0000 7fd56f46b640 1 -- 172.21.15.1:0/2783218375 wait complete. 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.234+0000 7fd56f46b640 1 Processor -- start 2024-03-24T10:36:37.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.234+0000 7fd56f46b640 1 -- start start 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5680b7d10 0x7fd5681e22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680d1e20 0x7fd5681e6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 0x7fd5681e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd5681e27e0 con 0x7fd5680d2540 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fd5680c9600 con 0x7fd5680b7d10 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56f46b640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fd5680c98e0 con 0x7fd5680d1e20 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 0x7fd5681e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 0x7fd5681e3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35458/0 (socket says 172.21.15.1:35458) 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 -- 172.21.15.1:0/1008890334 learned_addr learned my addr 172.21.15.1:0/1008890334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56d207640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5680b7d10 0x7fd5681e22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56ca06640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680d1e20 0x7fd5681e6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 -- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680d1e20 msgr2=0x7fd5681e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680d1e20 0x7fd5681e6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 -- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5680b7d10 msgr2=0x7fd5681e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5680b7d10 0x7fd5681e22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd558009400 con 0x7fd5680d2540 2024-03-24T10:36:37.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56d207640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5680b7d10 0x7fd5681e22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56ca06640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fd5680d1e20 0x7fd5681e6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.235+0000 7fd56da08640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 0x7fd5681e3a60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd560036040 tx=0x7fd560004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.236+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560009510 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.236+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56003e450 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.236+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd560009b30 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.236+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56003e820 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.236+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5680c9ea0 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.237+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5680cfaa0 con 0x7fd5680d2540 2024-03-24T10:36:37.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.237+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd56003e980 con 0x7fd5680d2540 2024-03-24T10:36:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd55dffb640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd54c06b810 0x7fd54c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fd5600ac9b0 con 0x7fd5680d2540 2024-03-24T10:36:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd56d207640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd54c06b810 0x7fd54c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd55dffb640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fd54c0716d0 0x7fd54c073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd54c0741a0 con 0x7fd54c0716d0 2024-03-24T10:36:37.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd5600b12c0 con 0x7fd5680d2540 2024-03-24T10:36:37.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd56d207640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd54c06b810 0x7fd54c06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564007d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.238+0000 7fd56ca06640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fd54c0716d0 0x7fd54c073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.239+0000 7fd56ca06640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fd54c0716d0 0x7fd54c073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.239+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== osd.6 v2:172.21.15.3:6808/2701673417 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd54c0741a0 con 0x7fd54c0716d0 2024-03-24T10:36:37.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 --> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd56804f470 con 0x7fd54c0716d0 2024-03-24T10:36:37.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd55dffb640 1 -- 172.21.15.1:0/1008890334 <== osd.6 v2:172.21.15.3:6808/2701673417 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd56804f470 con 0x7fd54c0716d0 2024-03-24T10:36:37.253 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:37.254 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:37.254 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:37.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fd54c0716d0 msgr2=0x7fd54c073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.3:6808/2701673417,v1:172.21.15.3:6809/2701673417] conn(0x7fd54c0716d0 0x7fd54c073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd54c06b810 msgr2=0x7fd54c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fd54c06b810 0x7fd54c06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564007d60 comp rx=0 tx=0).stop 2024-03-24T10:36:37.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 msgr2=0x7fd5681e3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56f46b640 1 --2- 172.21.15.1:0/1008890334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5680d2540 0x7fd5681e3a60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd560036040 tx=0x7fd560004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.253+0000 7fd56da08640 1 -- 172.21.15.1:0/1008890334 reap_dead start 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.254+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 shutdown_connections 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.254+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 >> 172.21.15.1:0/1008890334 conn(0x7fd56809d1d0 msgr2=0x7fd5680d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.254+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 shutdown_connections 2024-03-24T10:36:37.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.254+0000 7fd56f46b640 1 -- 172.21.15.1:0/1008890334 wait complete. 2024-03-24T10:36:37.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-24T10:36:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.366+0000 7f4b39197640 1 Processor -- start 2024-03-24T10:36:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.366+0000 7f4b39197640 1 -- start start 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b39197640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 0x7f4b3406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b39197640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4b340ae380 0x7f4b340b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b39197640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b341553c0 con 0x7f4b3406b5e0 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b39197640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f4b3406d250 con 0x7f4b340ae380 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b39197640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3406c7e0 con 0x7f4b34154fe0 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4b340ae380 0x7f4b340b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b32d9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 0x7f4b3406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4b340ae380 0x7f4b340b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:53906/0 (socket says 172.21.15.1:53906) 2024-03-24T10:36:37.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 -- 172.21.15.1:0/2126933297 learned_addr learned my addr 172.21.15.1:0/2126933297 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 -- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 msgr2=0x7f4b3406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:37.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 0x7f4b3406b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 -- 172.21.15.1:0/2126933297 >> v1:172.21.15.3:6789/0 conn(0x7f4b3406b5e0 legacy=0x7f4b340adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b3359e640 1 -- 172.21.15.1:0/2126933297 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3406bbd0 con 0x7f4b340ae380 2024-03-24T10:36:37.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.367+0000 7f4b32d9d640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 0x7f4b3406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:37.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.368+0000 7f4b3359e640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b340b3d00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009e50 tx=0x7f4b2c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e240b3b882c0873 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.368+0000 7f4b3259c640 1 -- 172.21.15.1:0/2126933297 <== mon.1 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c02cb00 con 0x7f4b340ae380 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.368+0000 7f4b3259c640 1 -- 172.21.15.1:0/2126933297 <== mon.1 v2:172.21.15.3:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2c02cc60 con 0x7f4b340ae380 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.368+0000 7f4b3259c640 1 -- 172.21.15.1:0/2126933297 <== mon.1 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c0343f0 con 0x7f4b340ae380 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 -- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 msgr2=0x7f4b340b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b340b3d00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009e50 tx=0x7f4b2c02bf60 comp rx=0 tx=0).stop 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 -- 172.21.15.1:0/2126933297 shutdown_connections 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b340b3d00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 --2- 172.21.15.1:0/2126933297 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b34154fe0 0x7f4b3406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 -- 172.21.15.1:0/2126933297 >> 172.21.15.1:0/2126933297 conn(0x7f4b340d0570 msgr2=0x7f4b340d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 -- 172.21.15.1:0/2126933297 shutdown_connections 2024-03-24T10:36:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.369+0000 7f4b39197640 1 -- 172.21.15.1:0/2126933297 wait complete. 2024-03-24T10:36:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.370+0000 7f4b39197640 1 Processor -- start 2024-03-24T10:36:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 -- start start 2024-03-24T10:36:37.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b3406b5e0 0x7f4b340cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b3414c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3406e050 con 0x7f4b34154fe0 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3406d440 con 0x7f4b340ae380 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b39197640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3406d720 con 0x7f4b3406b5e0 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b3359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b3359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35470/0 (socket says 172.21.15.1:35470) 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.371+0000 7f4b3359e640 1 -- 172.21.15.1:0/3507694148 learned_addr learned my addr 172.21.15.1:0/3507694148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b32d9d640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b3406b5e0 0x7f4b340cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b2a59c640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b3414c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 -- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b3406b5e0 msgr2=0x7f4b340cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b3406b5e0 0x7f4b340cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 -- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 msgr2=0x7f4b3414c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b3414c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 -- 172.21.15.1:0/3507694148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b24009400 con 0x7f4b34154fe0 2024-03-24T10:36:37.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b3359e640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c036040 tx=0x7f4b2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c009510 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2c03e450 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2c009b30 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c03e820 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.372+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3406dce0 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.373+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b3406e1c0 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.374+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4b2c03e980 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.374+0000 7f4b2bfff640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.374+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f4b2c0acce0 con 0x7f4b34154fe0 2024-03-24T10:36:37.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.374+0000 7f4b32d9d640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.375+0000 7f4b32d9d640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 0x7f4b1406dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009c60 tx=0x7f4b24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.378+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4b2c0778f0 con 0x7f4b34154fe0 2024-03-24T10:36:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.563+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f4b3406c470 con 0x7f4b1406b810 2024-03-24T10:36:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.563+0000 7f4b2bfff640 1 -- 172.21.15.1:0/3507694148 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4b240098b0 con 0x7f4b1406b810 2024-03-24T10:36:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-03-24T10:36:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 msgr2=0x7f4b1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 0x7f4b1406dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009c60 tx=0x7f4b24008040 comp rx=0 tx=0).stop 2024-03-24T10:36:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 msgr2=0x7f4b3406e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c036040 tx=0x7f4b2c004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 shutdown_connections 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b34154fe0 0x7f4b3406e4e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b340ae380 0x7f4b3414c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 --2- 172.21.15.1:0/3507694148 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f4b3406b5e0 0x7f4b340cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.565+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 >> 172.21.15.1:0/3507694148 conn(0x7f4b340d0570 msgr2=0x7f4b340afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.566+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 shutdown_connections 2024-03-24T10:36:37.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.566+0000 7f4b39197640 1 -- 172.21.15.1:0/3507694148 wait complete. 2024-03-24T10:36:37.577 INFO:tasks.ceph:Scrubbing osd.7 2024-03-24T10:36:37.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:36:37.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.682+0000 7f0fa140e640 1 Processor -- start 2024-03-24T10:36:37.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.682+0000 7f0fa140e640 1 -- start start 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0fa140e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c0d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0fa140e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9c0b7570 con 0x7f0f9c0d0920 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0fa140e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9c0b6f60 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0fa140e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9c0d3010 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0f9affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0f9c0d0920 0x7f0f9c0d2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42776/0 (socket says 172.21.15.1:42776) 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0f9affd640 1 -- 172.21.15.1:0/922333307 learned_addr learned my addr 172.21.15.1:0/922333307 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:37.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0f9a7fc640 1 --2- 172.21.15.1:0/922333307 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c0d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319457478 0 0) 0x7f0f9c0b7570 con 0x7f0f9c0d0920 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.683+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f88003520 con 0x7f0f9c0d0920 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166423301 0 0) 0x7f0f9c0b6f60 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f9c0b7570 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f9a7fc640 1 -- 172.21.15.1:0/922333307 >> v1:172.21.15.3:6789/0 conn(0x7f0f9c0b7a00 legacy=0x7f0f9c0b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f9a7fc640 1 -- 172.21.15.1:0/922333307 >> v1:172.21.15.1:6789/0 conn(0x7f0f9c0d0920 legacy=0x7f0f9c0d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1545163177 0 0) 0x7f0f88003520 con 0x7f0f9c0d0920 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.684+0000 7f0f9a7fc640 1 -- 172.21.15.1:0/922333307 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9c0d2d30 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0f9a7fc640 1 --2- 172.21.15.1:0/922333307 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c0d1180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0f840097d0 tx=0x7f0f8402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=938acb63ac351825 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94001670 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8402e050 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0f99ffb640 1 -- 172.21.15.1:0/922333307 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84030710 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 -- 172.21.15.1:0/922333307 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 msgr2=0x7f0f9c0d1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 --2- 172.21.15.1:0/922333307 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c0d1180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0f840097d0 tx=0x7f0f8402b800 comp rx=0 tx=0).stop 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 -- 172.21.15.1:0/922333307 shutdown_connections 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 --2- 172.21.15.1:0/922333307 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c0d1180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 -- 172.21.15.1:0/922333307 >> 172.21.15.1:0/922333307 conn(0x7f0f9c0d3f70 msgr2=0x7f0f9c0d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 -- 172.21.15.1:0/922333307 shutdown_connections 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.685+0000 7f0fa140e640 1 -- 172.21.15.1:0/922333307 wait complete. 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 Processor -- start 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 -- start start 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 0x7f0f9c1f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0d0920 0x7f0f9c0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c1593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9c0ca7d0 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9c1f48c0 con 0x7f0f9c0d0920 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0fa140e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9c1f3cd0 con 0x7f0f9c0d0d00 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0f9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 0x7f0f9c1f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0f9affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 0x7f0f9c1f7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35476/0 (socket says 172.21.15.1:35476) 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.687+0000 7f0f9affd640 1 -- 172.21.15.1:0/3879424060 learned_addr learned my addr 172.21.15.1:0/3879424060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9b7fe640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c1593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9a7fc640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0d0920 0x7f0f9c0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 -- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 msgr2=0x7f0f9c1593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f0f9c0d0d00 0x7f0f9c1593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 -- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0d0920 msgr2=0x7f0f9c0ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0d0920 0x7f0f9c0ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f84009400 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f9affd640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 0x7f0f9c1f7d40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c750 tx=0x7f0f9400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94005ad0 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f94005c30 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94007070 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f9c1f3f90 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.688+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f9c1f4550 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.689+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f9c0d16c0 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f940071d0 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f0f940894d0 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f9a7fc640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0f7c0716d0 0x7f0f7c073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f7c0741a0 con 0x7f0f7c0716d0 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f0f9408bb30 con 0x7f0f9c0b7a00 2024-03-24T10:36:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f9b7fe640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0f7c0716d0 0x7f0f7c073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.690+0000 7f0f9a7fc640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0f84034000 tx=0x7f0f84008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.691+0000 7f0f9b7fe640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0f7c0716d0 0x7f0f7c073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.692+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== osd.7 v2:172.21.15.3:6816/1606588586 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f7c0741a0 con 0x7f0f7c0716d0 2024-03-24T10:36:37.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.705+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 --> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0f9c1f4a00 con 0x7f0f7c0716d0 2024-03-24T10:36:37.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.705+0000 7f0f937fe640 1 -- 172.21.15.1:0/3879424060 <== osd.7 v2:172.21.15.3:6816/1606588586 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0f8c001e30 con 0x7f0f7c0716d0 2024-03-24T10:36:37.706 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-24T10:36:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:36:37.706 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-24T10:36:37.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0f7c0716d0 msgr2=0x7f0f7c073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.3:6816/1606588586,v1:172.21.15.3:6817/1606588586] conn(0x7f0f7c0716d0 0x7f0f7c073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0f7c06b810 msgr2=0x7f0f7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0f84034000 tx=0x7f0f84008030 comp rx=0 tx=0).stop 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 msgr2=0x7f0f9c1f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 --2- 172.21.15.1:0/3879424060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f9c0b7a00 0x7f0f9c1f7d40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c750 tx=0x7f0f9400cc10 comp rx=0 tx=0).stop 2024-03-24T10:36:37.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0f9b7fe640 1 -- 172.21.15.1:0/3879424060 reap_dead start 2024-03-24T10:36:37.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 shutdown_connections 2024-03-24T10:36:37.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 >> 172.21.15.1:0/3879424060 conn(0x7f0f9c0d3f70 msgr2=0x7f0f9c15d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 shutdown_connections 2024-03-24T10:36:37.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.706+0000 7f0fa140e640 1 -- 172.21.15.1:0/3879424060 wait complete. 2024-03-24T10:36:37.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-24T10:36:37.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.818+0000 7f5309125640 1 Processor -- start 2024-03-24T10:36:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.818+0000 7f5309125640 1 -- start start 2024-03-24T10:36:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5309125640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f530415e390 0x7f53040d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5309125640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530415e770 con 0x7f53040b4ba0 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5309125640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53040d4e40 con 0x7f53040d1000 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5309125640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f53040b3190 con 0x7f530415e390 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5302d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f530415e390 0x7f53040d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f530359e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53040b4ba0 0x7f53040cf170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42790/0 (socket says 172.21.15.1:42790) 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5302d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f530415e390 0x7f53040d24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35480/0 (socket says 172.21.15.1:35480) 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f530359e640 1 -- 172.21.15.1:0/979285845 learned_addr learned my addr 172.21.15.1:0/979285845 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5302d9d640 1 -- 172.21.15.1:0/979285845 >> v1:172.21.15.3:6789/0 conn(0x7f53040d1000 legacy=0x7f53040cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:37.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5302d9d640 1 -- 172.21.15.1:0/979285845 >> v1:172.21.15.1:6789/0 conn(0x7f53040b4ba0 legacy=0x7f53040cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5301d9b640 1 -- 172.21.15.1:0/979285845 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333448603 0 0) 0x7f52f4001490 con 0x7f53040b4ba0 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.819+0000 7f5302d9d640 1 -- 172.21.15.1:0/979285845 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53040d1e40 con 0x7f530415e390 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.820+0000 7f5302d9d640 1 --2- 172.21.15.1:0/979285845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f530415e390 0x7f53040d24a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0097a0 tx=0x7f52fc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f83c1aea870045 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.820+0000 7f5301d9b640 1 -- 172.21.15.1:0/979285845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc02f050 con 0x7f530415e390 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.820+0000 7f5301d9b640 1 -- 172.21.15.1:0/979285845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52fc031700 con 0x7f530415e390 2024-03-24T10:36:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.820+0000 7f5301d9b640 1 -- 172.21.15.1:0/979285845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc031a90 con 0x7f530415e390 2024-03-24T10:36:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 -- 172.21.15.1:0/979285845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f530415e390 msgr2=0x7f53040d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 --2- 172.21.15.1:0/979285845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f530415e390 0x7f53040d24a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0097a0 tx=0x7f52fc02b7a0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 -- 172.21.15.1:0/979285845 shutdown_connections 2024-03-24T10:36:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 --2- 172.21.15.1:0/979285845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f530415e390 0x7f53040d24a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 -- 172.21.15.1:0/979285845 >> 172.21.15.1:0/979285845 conn(0x7f53040d3ec0 msgr2=0x7f53040d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 -- 172.21.15.1:0/979285845 shutdown_connections 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.821+0000 7f5309125640 1 -- 172.21.15.1:0/979285845 wait complete. 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.822+0000 7f5309125640 1 Processor -- start 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.822+0000 7f5309125640 1 -- start start 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53040b4ba0 0x7f5304152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f530415e390 0x7f53040b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f53040cc1b0 con 0x7f53040d1000 2024-03-24T10:36:37.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f53040cb530 con 0x7f530415e390 2024-03-24T10:36:37.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5309125640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f53040cb810 con 0x7f53040b4ba0 2024-03-24T10:36:37.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f530259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f530259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35482/0 (socket says 172.21.15.1:35482) 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f530259c640 1 -- 172.21.15.1:0/4125422872 learned_addr learned my addr 172.21.15.1:0/4125422872 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f530359e640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f530415e390 0x7f53040b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.823+0000 7f5302d9d640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53040b4ba0 0x7f5304152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 -- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53040b4ba0 msgr2=0x7f5304152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53040b4ba0 0x7f5304152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 -- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f530415e390 msgr2=0x7f53040b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f530415e390 0x7f53040b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:37.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 -- 172.21.15.1:0/4125422872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc009400 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f530259c640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b3c0 tx=0x7f52f800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011830 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f8011990 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53040cbaf0 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011d80 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.824+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5304152880 con 0x7f53040d1000 2024-03-24T10:36:37.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.825+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53040cc2f0 con 0x7f53040d1000 2024-03-24T10:36:37.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.825+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f52f801b460 con 0x7f53040d1000 2024-03-24T10:36:37.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.826+0000 7f52eb7fe640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 0x7f52f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:37.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.826+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f52f8089830 con 0x7f53040d1000 2024-03-24T10:36:37.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.826+0000 7f5302d9d640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 0x7f52f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:37.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.826+0000 7f5302d9d640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 0x7f52f006dcd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f52fc035000 tx=0x7f52fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:37.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:37.830+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f53040cc2f0 con 0x7f53040d1000 2024-03-24T10:36:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.009+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f53040cf880 con 0x7f52f006b810 2024-03-24T10:36:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.009+0000 7f52eb7fe640 1 -- 172.21.15.1:0/4125422872 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f52fc031ad0 con 0x7f52f006b810 2024-03-24T10:36:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-03-24T10:36:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 msgr2=0x7f52f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 0x7f52f006dcd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f52fc035000 tx=0x7f52fc008030 comp rx=0 tx=0).stop 2024-03-24T10:36:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 msgr2=0x7f53040cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f52f800b3c0 tx=0x7f52f800b880 comp rx=0 tx=0).stop 2024-03-24T10:36:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 shutdown_connections 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f52f006b810 0x7f52f006dcd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f530415e390 0x7f53040b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f53040d1000 0x7f53040cc5d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 --2- 172.21.15.1:0/4125422872 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f53040b4ba0 0x7f5304152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 >> 172.21.15.1:0/4125422872 conn(0x7f53040d3ec0 msgr2=0x7f53040d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 shutdown_connections 2024-03-24T10:36:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.011+0000 7f5309125640 1 -- 172.21.15.1:0/4125422872 wait complete. 2024-03-24T10:36:38.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:36:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.123+0000 7fdc4635e640 1 Processor -- start 2024-03-24T10:36:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.123+0000 7fdc4635e640 1 -- start start 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc4635e640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 0x7fdc4006b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc4635e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4006b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc4635e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc40072c10 con 0x7fdc40072710 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc4635e640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7fdc4006dde0 con 0x7fdc4015e0a0 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc4635e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4006e0c0 con 0x7fdc400b0180 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc3ffff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 0x7fdc4006b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc448fb640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4006b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc3ffff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 0x7fdc4006b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:53934/0 (socket says 172.21.15.1:53934) 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.125+0000 7fdc3ffff640 1 -- 172.21.15.1:0/1563422087 learned_addr learned my addr 172.21.15.1:0/1563422087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.124+0000 7fdc3f7fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdc40072710 0x7fdc40161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:43782/0 (socket says 172.21.15.1:43782) 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.125+0000 7fdc448fb640 1 -- 172.21.15.1:0/1563422087 >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 msgr2=0x7fdc4006b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.125+0000 7fdc448fb640 1 --2- 172.21.15.1:0/1563422087 >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 0x7fdc4006b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.125+0000 7fdc448fb640 1 -- 172.21.15.1:0/1563422087 >> v1:172.21.15.3:6789/0 conn(0x7fdc40072710 legacy=0x7fdc40161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.125+0000 7fdc448fb640 1 -- 172.21.15.1:0/1563422087 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4006e3a0 con 0x7fdc400b0180 2024-03-24T10:36:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc448fb640 1 --2- 172.21.15.1:0/1563422087 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4006b5e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009d90 tx=0x7fdc3802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fa2fa180b1943e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc3effd640 1 -- 172.21.15.1:0/1563422087 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3802cc50 con 0x7fdc400b0180 2024-03-24T10:36:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc3effd640 1 -- 172.21.15.1:0/1563422087 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3802cdb0 con 0x7fdc400b0180 2024-03-24T10:36:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc3effd640 1 -- 172.21.15.1:0/1563422087 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38034660 con 0x7fdc400b0180 2024-03-24T10:36:38.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 -- 172.21.15.1:0/1563422087 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 msgr2=0x7fdc4006b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 --2- 172.21.15.1:0/1563422087 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4006b5e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009d90 tx=0x7fdc3802bea0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 -- 172.21.15.1:0/1563422087 shutdown_connections 2024-03-24T10:36:38.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 --2- 172.21.15.1:0/1563422087 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4006b5e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 --2- 172.21.15.1:0/1563422087 >> v2:172.21.15.3:3300/0 conn(0x7fdc4015e0a0 0x7fdc4006b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 -- 172.21.15.1:0/1563422087 >> 172.21.15.1:0/1563422087 conn(0x7fdc400b2820 msgr2=0x7fdc400b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 -- 172.21.15.1:0/1563422087 shutdown_connections 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.126+0000 7fdc4635e640 1 -- 172.21.15.1:0/1563422087 wait complete. 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 Processor -- start 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 -- start start 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4015e0a0 0x7fdc401f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc401f8150 con 0x7fdc40072710 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7fdc401f7540 con 0x7fdc4015e0a0 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.128+0000 7fdc4635e640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7fdc401f7820 con 0x7fdc400b0180 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35494/0 (socket says 172.21.15.1:35494) 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 -- 172.21.15.1:0/4230707522 learned_addr learned my addr 172.21.15.1:0/4230707522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3f7fe640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc448fb640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4015e0a0 0x7fdc401f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 -- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 msgr2=0x7fdc4015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4015ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 -- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4015e0a0 msgr2=0x7fdc401f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4015e0a0 0x7fdc401f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 -- 172.21.15.1:0/4230707522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc38009a70 con 0x7fdc40072710 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3ffff640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000e840 tx=0x7fdc3000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000c8e0 con 0x7fdc40072710 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3000ca40 con 0x7fdc40072710 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc401f7b00 con 0x7fdc40072710 2024-03-24T10:36:38.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000cdf0 con 0x7fdc40072710 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.129+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc401f7210 con 0x7fdc40072710 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.130+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4004f4e0 con 0x7fdc40072710 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.131+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdc30013470 con 0x7fdc40072710 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.131+0000 7fdc3cff9640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.131+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7fdc30089e30 con 0x7fdc40072710 2024-03-24T10:36:38.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.131+0000 7fdc3f7fe640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:38.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.132+0000 7fdc3f7fe640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdc401f7170 tx=0x7fdc34007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:38.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.135+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdc30054c90 con 0x7fdc40072710 2024-03-24T10:36:38.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.314+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc40157080 con 0x7fdc2c06b810 2024-03-24T10:36:38.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.314+0000 7fdc3cff9640 1 -- 172.21.15.1:0/4230707522 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13449 (secure 0 0 0) 0x7fdc40157080 con 0x7fdc2c06b810 2024-03-24T10:36:38.315 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:38.315 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 msgr2=0x7fdc2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdc401f7170 tx=0x7fdc34007e80 comp rx=0 tx=0).stop 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 msgr2=0x7fdc4015c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000e840 tx=0x7fdc3000ed00 comp rx=0 tx=0).stop 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 shutdown_connections 2024-03-24T10:36:38.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7fdc2c06b810 0x7fdc2c06dcd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4015e0a0 0x7fdc401f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7fdc400b0180 0x7fdc4015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 --2- 172.21.15.1:0/4230707522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc40072710 0x7fdc4015c950 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.316+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 >> 172.21.15.1:0/4230707522 conn(0x7fdc400b2820 msgr2=0x7fdc400b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.317+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 shutdown_connections 2024-03-24T10:36:38.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:38.317+0000 7fdc4635e640 1 -- 172.21.15.1:0/4230707522 wait complete. 2024-03-24T10:36:38.327 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-03-24T10:36:36.546884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238948,"kb_used_data":6136,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735772,"statfs":{"total":773094113280,"available":772849430528,"internally_reserved":0,"allocated":6283264,"data_stored":3292091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-71,"num_write_kb":-42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001291"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":231,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T10:36:31.128230+0000","last_change":"2024-03-24T10:33:44.073863+0000","last_active":"2024-03-24T10:36:31.128230+0000","last_peered":"2024-03-24T10:36:31.128230+0000","last_clean":"2024-03-24T10:36:31.128230+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:36:31.128230+0000","last_undegraded":"2024-03-24T10:36:31.128230+0000","last_fullsized":"2024-03-24T10:36:31.128230+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:43.004374+0000","last_clean_scrub_stamp":"2024-03-24T10:33:43.004374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:25.007245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1216512,"data_stored":849418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1232896,"data_stored":854983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:36:38.330 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-24T10:33:43.004374+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=33, tm_sec=43, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=36, tm_sec=33, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:36:38.330 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-24T10:36:56.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:36:56.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:36:58.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:36:58.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.444+0000 7f367290c640 1 Processor -- start 2024-03-24T10:36:58.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.444+0000 7f367290c640 1 -- start start 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f367290c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f366c06d020 0x7f366c072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f367290c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f366c15b9c0 con 0x7f366c15b470 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f367290c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f366c070ca0 con 0x7f366c1549e0 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f367290c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f366c0d0690 con 0x7f366c06d020 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f3670ea9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f366c06d020 0x7f366c072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f366b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f366c15b470 0x7f366c06c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40724/0 (socket says 172.21.15.1:40724) 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f3670ea9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f366c06d020 0x7f366c072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33410/0 (socket says 172.21.15.1:33410) 2024-03-24T10:36:58.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f366b7fe640 1 -- 172.21.15.1:0/1663278306 learned_addr learned my addr 172.21.15.1:0/1663278306 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f3670ea9640 1 -- 172.21.15.1:0/1663278306 >> v1:172.21.15.3:6789/0 conn(0x7f366c1549e0 legacy=0x7f366c071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f3670ea9640 1 -- 172.21.15.1:0/1663278306 >> v1:172.21.15.1:6789/0 conn(0x7f366c15b470 legacy=0x7f366c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f366affd640 1 -- 172.21.15.1:0/1663278306 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041349175 0 0) 0x7f366c15b9c0 con 0x7f366c15b470 2024-03-24T10:36:58.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.445+0000 7f3670ea9640 1 -- 172.21.15.1:0/1663278306 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c070f80 con 0x7f366c06d020 2024-03-24T10:36:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.446+0000 7f3670ea9640 1 --2- 172.21.15.1:0/1663278306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c06d020 0x7f366c072990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3664009d90 tx=0x7f366402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ba2a82e992b47a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.446+0000 7f366affd640 1 -- 172.21.15.1:0/1663278306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366402cd00 con 0x7f366c06d020 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.446+0000 7f366affd640 1 -- 172.21.15.1:0/1663278306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366402ce60 con 0x7f366c06d020 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.446+0000 7f366affd640 1 -- 172.21.15.1:0/1663278306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664034660 con 0x7f366c06d020 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 -- 172.21.15.1:0/1663278306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c06d020 msgr2=0x7f366c072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 --2- 172.21.15.1:0/1663278306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c06d020 0x7f366c072990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3664009d90 tx=0x7f366402bf50 comp rx=0 tx=0).stop 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 -- 172.21.15.1:0/1663278306 shutdown_connections 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 --2- 172.21.15.1:0/1663278306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c06d020 0x7f366c072990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 -- 172.21.15.1:0/1663278306 >> 172.21.15.1:0/1663278306 conn(0x7f366c0b17a0 msgr2=0x7f366c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 -- 172.21.15.1:0/1663278306 shutdown_connections 2024-03-24T10:36:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.447+0000 7f367290c640 1 -- 172.21.15.1:0/1663278306 wait complete. 2024-03-24T10:36:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.448+0000 7f367290c640 1 Processor -- start 2024-03-24T10:36:58.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.448+0000 7f367290c640 1 -- start start 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f366c1549e0 0x7f366c1388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 0x7f366c138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f366c152dc0 con 0x7f366c15b470 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1530a0 con 0x7f366c13df70 2024-03-24T10:36:58.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f367290c640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f366c142ba0 con 0x7f366c1549e0 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33414/0 (socket says 172.21.15.1:33414) 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 -- 172.21.15.1:0/1153007462 learned_addr learned my addr 172.21.15.1:0/1153007462 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f3670ea9640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 0x7f366c138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 -- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f366c1549e0 msgr2=0x7f366c1388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366bfff640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f366c1549e0 0x7f366c1388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f366c1549e0 0x7f366c1388e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 -- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 msgr2=0x7f366c138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 0x7f366c138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f366b7fe640 1 -- 172.21.15.1:0/1153007462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3664009a70 con 0x7f366c15b470 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.449+0000 7f3670ea9640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 0x7f366c138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f366b7fe640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f365c00c4a0 tx=0x7f365c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c0074e0 con 0x7f366c15b470 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f365c007640 con 0x7f366c15b470 2024-03-24T10:36:58.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366c142e80 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c004a00 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.450+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366c152b50 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.451+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366c04f4e0 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.451+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f365c004b60 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.452+0000 7f3668ff9640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 0x7f365806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.452+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f365c089400 con 0x7f366c15b470 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.452+0000 7f366bfff640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 0x7f365806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.452+0000 7f366bfff640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 0x7f365806dcd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3660000f60 tx=0x7f3660009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.456+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f365c054120 con 0x7f366c15b470 2024-03-24T10:36:58.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.634+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 --> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f366c071540 con 0x7f365806b810 2024-03-24T10:36:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.635+0000 7f3668ff9640 1 -- 172.21.15.1:0/1153007462 <== mgr.4107 v2:172.21.15.1:6832/2979692092 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f3660001390 con 0x7f365806b810 2024-03-24T10:36:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:58.636 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-24T10:36:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.636+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 msgr2=0x7f365806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.636+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 0x7f365806dcd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3660000f60 tx=0x7f3660009040 comp rx=0 tx=0).stop 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.636+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 msgr2=0x7f366c13da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.636+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f365c00c4a0 tx=0x7f365c00c960 comp rx=0 tx=0).stop 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 shutdown_connections 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f365806b810 0x7f365806dcd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f366c13df70 0x7f366c138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f366c15b470 0x7f366c13da30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 --2- 172.21.15.1:0/1153007462 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f366c1549e0 0x7f366c1388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 >> 172.21.15.1:0/1153007462 conn(0x7f366c0b17a0 msgr2=0x7f366c06dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 shutdown_connections 2024-03-24T10:36:58.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.637+0000 7f367290c640 1 -- 172.21.15.1:0/1153007462 wait complete. 2024-03-24T10:36:58.649 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-03-24T10:36:58.551123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238956,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735764,"statfs":{"total":773094113280,"available":772849422336,"internally_reserved":0,"allocated":6299648,"data_stored":3297656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"1.0","version":"53'119","reported_seq":242,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T10:36:38.717313+0000","last_change":"2024-03-24T10:36:38.717371+0000","last_active":"2024-03-24T10:36:38.717313+0000","last_peered":"2024-03-24T10:36:38.717313+0000","last_clean":"2024-03-24T10:36:38.717313+0000","last_became_active":"2024-03-24T10:33:44.072737+0000","last_became_peered":"2024-03-24T10:33:44.072737+0000","last_unstale":"2024-03-24T10:36:38.717313+0000","last_undegraded":"2024-03-24T10:36:38.717313+0000","last_fullsized":"2024-03-24T10:36:38.717313+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'119","last_scrub_stamp":"2024-03-24T10:36:38.717217+0000","last_deep_scrub":"53'119","last_deep_scrub_stamp":"2024-03-24T10:36:38.717217+0000","last_clean_scrub_stamp":"2024-03-24T10:36:38.717217+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:08.588658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1232896,"data_stored":854983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1232896,"data_stored":854983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:36:58.650 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-24T10:36:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.750+0000 7f387aec9640 1 Processor -- start 2024-03-24T10:36:58.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.750+0000 7f387aec9640 1 -- start start 2024-03-24T10:36:58.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f387aec9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f387415e860 0x7f38740cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f387aec9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 0x7f387406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f387aec9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38740b45c0 con 0x7f38740cbf90 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f387aec9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f387406c930 con 0x7f387415e860 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f387aec9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f38740d0970 con 0x7f38740d04c0 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f387415e860 0x7f38740cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3878c65640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f38740cbf90 0x7f387415bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40738/0 (socket says 172.21.15.1:40738) 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f387415e860 0x7f38740cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33428/0 (socket says 172.21.15.1:33428) 2024-03-24T10:36:58.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3878c65640 1 -- 172.21.15.1:0/4286236697 learned_addr learned my addr 172.21.15.1:0/4286236697 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 -- 172.21.15.1:0/4286236697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 msgr2=0x7f387406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3879466640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 0x7f387406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 0x7f387406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f38737fe640 1 -- 172.21.15.1:0/4286236697 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2784884357 0 0) 0x7f38740b45c0 con 0x7f38740cbf90 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 -- 172.21.15.1:0/4286236697 >> v1:172.21.15.1:6789/0 conn(0x7f38740cbf90 legacy=0x7f387415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3873fff640 1 -- 172.21.15.1:0/4286236697 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387415ed00 con 0x7f387415e860 2024-03-24T10:36:58.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.751+0000 7f3879466640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 0x7f387406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:36:58.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f3873fff640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387415e860 0x7f38740cff80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3864009dc0 tx=0x7f386402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece8416ad4a06f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f38737fe640 1 -- 172.21.15.1:0/4286236697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386402cbb0 con 0x7f387415e860 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f38737fe640 1 -- 172.21.15.1:0/4286236697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386402cd10 con 0x7f387415e860 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f38737fe640 1 -- 172.21.15.1:0/4286236697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864034660 con 0x7f387415e860 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f387aec9640 1 -- 172.21.15.1:0/4286236697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387415e860 msgr2=0x7f38740cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f387aec9640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387415e860 0x7f38740cff80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3864009dc0 tx=0x7f386402be00 comp rx=0 tx=0).stop 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f387aec9640 1 -- 172.21.15.1:0/4286236697 shutdown_connections 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f387aec9640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f38740d04c0 0x7f387406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.752+0000 7f387aec9640 1 --2- 172.21.15.1:0/4286236697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387415e860 0x7f38740cff80 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.753+0000 7f387aec9640 1 -- 172.21.15.1:0/4286236697 >> 172.21.15.1:0/4286236697 conn(0x7f387406b4f0 msgr2=0x7f387406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.753+0000 7f387aec9640 1 -- 172.21.15.1:0/4286236697 shutdown_connections 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.753+0000 7f387aec9640 1 -- 172.21.15.1:0/4286236697 wait complete. 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 Processor -- start 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 -- start start 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 0x7f38741ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f387415e860 0x7f38740cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f38740cdb00 con 0x7f38740cbf90 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f387414ff60 con 0x7f38740d04c0 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.754+0000 7f387aec9640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f3874159dc0 con 0x7f387415e860 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33432/0 (socket says 172.21.15.1:33432) 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 -- 172.21.15.1:0/802291614 learned_addr learned my addr 172.21.15.1:0/802291614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3879466640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f387415e860 0x7f38740cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3873fff640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 0x7f38741ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 -- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f387415e860 msgr2=0x7f38740cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f387415e860 0x7f38740cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 -- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 msgr2=0x7f38741ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 0x7f38741ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 -- 172.21.15.1:0/802291614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3864009a70 con 0x7f38740cbf90 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3873fff640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 0x7f38741ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f3878c65640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f386c00c4a0 tx=0x7f386c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c005ce0 con 0x7f38740cbf90 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386c005e40 con 0x7f38740cbf90 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38740cd160 con 0x7f38740cbf90 2024-03-24T10:36:58.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c0072b0 con 0x7f38740cbf90 2024-03-24T10:36:58.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.755+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38740cd720 con 0x7f38740cbf90 2024-03-24T10:36:58.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.756+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38740cdc40 con 0x7f38740cbf90 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.757+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f386c007410 con 0x7f38740cbf90 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.757+0000 7f38717fa640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 0x7f385406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.757+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f386c089440 con 0x7f38740cbf90 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.757+0000 7f3873fff640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 0x7f385406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.758+0000 7f3873fff640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 0x7f385406dcd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3864036000 tx=0x7f3864030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:58.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.762+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f386c054200 con 0x7f38740cbf90 2024-03-24T10:36:58.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.944+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f387404f470 con 0x7f38740cbf90 2024-03-24T10:36:58.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.947+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f386c0578c0 con 0x7f38740cbf90 2024-03-24T10:36:58.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.948+0000 7f38717fa640 1 -- 172.21.15.1:0/802291614 <== mon.0 v2:172.21.15.1:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f386c0578c0 con 0x7f38740cbf90 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 msgr2=0x7f385406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 0x7f385406dcd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3864036000 tx=0x7f3864030000 comp rx=0 tx=0).stop 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 msgr2=0x7f387414fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f386c00c4a0 tx=0x7f386c00c960 comp rx=0 tx=0).stop 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 shutdown_connections 2024-03-24T10:36:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f385406b810 0x7f385406dcd0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f387415e860 0x7f38740cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38740d04c0 0x7f38741ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 --2- 172.21.15.1:0/802291614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38740cbf90 0x7f387414fa20 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.949+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 >> 172.21.15.1:0/802291614 conn(0x7f387406b4f0 msgr2=0x7f38740b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.950+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 shutdown_connections 2024-03-24T10:36:58.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:58.950+0000 7f387aec9640 1 -- 172.21.15.1:0/802291614 wait complete. 2024-03-24T10:36:58.962 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T10:36:58.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:36:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.062+0000 7f2792e9d640 1 Processor -- start 2024-03-24T10:36:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.062+0000 7f2792e9d640 1 -- start start 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2792e9d640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2792e9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 0x7f278c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2792e9d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278c0cdd60 con 0x7f278c144770 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2792e9d640 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 -- 0x7f278c06eaa0 con 0x7f278c0d00f0 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2792e9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f278c0b3470 con 0x7f278c0b2aa0 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f278bfff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f279143a640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 0x7f278c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2790c39640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f278c144770 0x7f278c06da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.1:53518/0 (socket says 172.21.15.1:53518) 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f278bfff640 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.1:55990/0 (socket says 172.21.15.1:55990) 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f2790c39640 1 -- 172.21.15.1:0/2183298926 learned_addr learned my addr 172.21.15.1:0/2183298926 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f279143a640 1 -- 172.21.15.1:0/2183298926 >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 msgr2=0x7f278c070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f279143a640 1 --2- 172.21.15.1:0/2183298926 >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.063+0000 7f279143a640 1 -- 172.21.15.1:0/2183298926 >> v1:172.21.15.3:6789/0 conn(0x7f278c144770 legacy=0x7f278c06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.064+0000 7f278b7fe640 1 -- 172.21.15.1:0/2183298926 <== mon.1 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255117280 0 0) 0x7f278c0cdd60 con 0x7f278c144770 2024-03-24T10:36:59.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.064+0000 7f279143a640 1 -- 172.21.15.1:0/2183298926 --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c06e200 con 0x7f278c0b2aa0 2024-03-24T10:36:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.064+0000 7f278bfff640 1 --2- 172.21.15.1:0/2183298926 >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:36:59.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.064+0000 7f279143a640 1 --2- 172.21.15.1:0/2183298926 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 0x7f278c0b2f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2784009d90 tx=0x7f278402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0fb7e02ba316d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:59.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f278b7fe640 1 -- 172.21.15.1:0/2183298926 <== mon.2 v2:172.21.15.3:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278402cc50 con 0x7f278c0b2aa0 2024-03-24T10:36:59.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f278b7fe640 1 -- 172.21.15.1:0/2183298926 <== mon.2 v2:172.21.15.3:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f278402cdb0 con 0x7f278c0b2aa0 2024-03-24T10:36:59.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f278b7fe640 1 -- 172.21.15.1:0/2183298926 <== mon.2 v2:172.21.15.3:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784034680 con 0x7f278c0b2aa0 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 -- 172.21.15.1:0/2183298926 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 msgr2=0x7f278c0b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 --2- 172.21.15.1:0/2183298926 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 0x7f278c0b2f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2784009d90 tx=0x7f278402bea0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 -- 172.21.15.1:0/2183298926 shutdown_connections 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 --2- 172.21.15.1:0/2183298926 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0b2aa0 0x7f278c0b2f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 --2- 172.21.15.1:0/2183298926 >> v2:172.21.15.3:3300/0 conn(0x7f278c0d00f0 0x7f278c070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 -- 172.21.15.1:0/2183298926 >> 172.21.15.1:0/2183298926 conn(0x7f278c0b17a0 msgr2=0x7f278c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 -- 172.21.15.1:0/2183298926 shutdown_connections 2024-03-24T10:36:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.065+0000 7f2792e9d640 1 -- 172.21.15.1:0/2183298926 wait complete. 2024-03-24T10:36:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.066+0000 7f2792e9d640 1 Processor -- start 2024-03-24T10:36:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.066+0000 7f2792e9d640 1 -- start start 2024-03-24T10:36:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f278c0b2aa0 0x7f278c13de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 --2- >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0d00f0 0x7f278c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f278c13f420 con 0x7f278c144770 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 -- 0x7f278c13e3a0 con 0x7f278c0b2aa0 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2792e9d640 1 -- --> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] -- mon_getmap magic: 0 -- 0x7f278c071de0 con 0x7f278c0d00f0 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f279143a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f279143a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33440/0 (socket says 172.21.15.1:33440) 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f279143a640 1 -- 172.21.15.1:0/3972470016 learned_addr learned my addr 172.21.15.1:0/3972470016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-24T10:36:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f2790c39640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f278c0b2aa0 0x7f278c13de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.067+0000 7f278bfff640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0d00f0 0x7f278c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 -- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0d00f0 msgr2=0x7f278c15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0d00f0 0x7f278c15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 -- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f278c0b2aa0 msgr2=0x7f278c13de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f278c0b2aa0 0x7f278c13de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 -- 172.21.15.1:0/3972470016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2778009400 con 0x7f278c144770 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f279143a640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2784037040 tx=0x7f2784004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784040650 con 0x7f278c144770 2024-03-24T10:36:59.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f27840407f0 con 0x7f278c144770 2024-03-24T10:36:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2784009a70 con 0x7f278c144770 2024-03-24T10:36:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784040bc0 con 0x7f278c144770 2024-03-24T10:36:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.068+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c0714b0 con 0x7f278c144770 2024-03-24T10:36:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.069+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c068680 con 0x7f278c144770 2024-03-24T10:36:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.069+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2784040d20 con 0x7f278c144770 2024-03-24T10:36:59.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.070+0000 7f27897fa640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 0x7f277006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:59.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.070+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4943+0+0 (secure 0 0 0) 0x7f27840acd40 con 0x7f278c144770 2024-03-24T10:36:59.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.070+0000 7f2790c39640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 0x7f277006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:59.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.070+0000 7f2790c39640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 0x7f277006dd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2780000f60 tx=0x7f2780007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:59.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.074+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2784077af0 con 0x7f278c144770 2024-03-24T10:36:59.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.293+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f278c071f50 con 0x7f278c144770 2024-03-24T10:36:59.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.293+0000 7f27897fa640 1 -- 172.21.15.1:0/3972470016 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3857 (secure 0 0 0) 0x7f2784039090 con 0x7f278c144770 2024-03-24T10:36:59.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-24T10:36:59.295 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6834/3119273888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":3119273888},{"type":"v1","addr":"172.21.15.3:6834","nonce":3119273888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5303,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3644953052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6833","nonce":3644953052},{"type":"v1","addr":"172.21.15.3:6835","nonce":3644953052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/1030623611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1030623611},{"type":"v1","addr":"172.21.15.1:6835","nonce":1030623611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6450,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/4107884196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":4107884196},{"type":"v1","addr":"172.21.15.1:6837","nonce":4107884196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-03-24T10:36:59.295 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 27 2024-03-24T10:36:59.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 msgr2=0x7f277006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 0x7f277006dd00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2780000f60 tx=0x7f2780007c00 comp rx=0 tx=0).stop 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 msgr2=0x7f278c072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2784037040 tx=0x7f2784004a40 comp rx=0 tx=0).stop 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 shutdown_connections 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:6832/2979692092,v1:172.21.15.1:6833/2979692092] conn(0x7f277006b840 0x7f277006dd00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f278c144770 0x7f278c072270 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] conn(0x7f278c0d00f0 0x7f278c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 --2- 172.21.15.1:0/3972470016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f278c0b2aa0 0x7f278c13de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:59.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 >> 172.21.15.1:0/3972470016 conn(0x7f278c0b17a0 msgr2=0x7f278c0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:59.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 shutdown_connections 2024-03-24T10:36:59.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-24T10:36:59.295+0000 7f2792e9d640 1 -- 172.21.15.1:0/3972470016 wait complete. 2024-03-24T10:36:59.307 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-24T10:36:59.307 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-24T10:36:59.308 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T10:36:59.308 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.317 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T10:36:59.317 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T10:36:59.317 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.367 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T10:36:59.367 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T10:36:59.368 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.380 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T10:36:59.380 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T10:36:59.380 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.432 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T10:36:59.433 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-24T10:36:59.433 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-24T10:36:59.433 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.459 INFO:tasks.ceph.osd.0:Stopped 2024-03-24T10:36:59.459 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-24T10:36:59.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.527 INFO:tasks.ceph.osd.1:Stopped 2024-03-24T10:36:59.527 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-24T10:36:59.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.596 INFO:tasks.ceph.osd.2:Stopped 2024-03-24T10:36:59.596 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-24T10:36:59.596 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.670 INFO:tasks.ceph.osd.3:Stopped 2024-03-24T10:36:59.670 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-24T10:36:59.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.709 INFO:tasks.ceph.osd.4:Stopped 2024-03-24T10:36:59.710 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-24T10:36:59.710 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.781 INFO:tasks.ceph.osd.5:Stopped 2024-03-24T10:36:59.781 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-24T10:36:59.781 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.852 INFO:tasks.ceph.osd.6:Stopped 2024-03-24T10:36:59.853 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-24T10:36:59.853 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.925 INFO:tasks.ceph.osd.7:Stopped 2024-03-24T10:36:59.925 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-24T10:36:59.925 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-24T10:36:59.925 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.953 INFO:tasks.ceph.mgr.y:Stopped 2024-03-24T10:36:59.953 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-24T10:36:59.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:36:59.993 INFO:tasks.ceph.mgr.x:Stopped 2024-03-24T10:36:59.994 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-24T10:36:59.994 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-24T10:36:59.994 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:37:00.006 INFO:tasks.ceph.mon.a:Stopped 2024-03-24T10:37:00.006 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-24T10:37:00.007 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:37:00.047 INFO:tasks.ceph.mon.b:Stopped 2024-03-24T10:37:00.048 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-24T10:37:00.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T10:37:00.101 INFO:tasks.ceph.mon.c:Stopped 2024-03-24T10:37:00.102 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-24T10:37:00.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-03-24T10:37:00.134 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:37:00.135 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-24T10:37:00.346 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:37:00.347 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-24T10:37:00.412 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:37:00.413 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-24T10:37:00.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:37:00.471 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-24T10:37:00.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:37:00.520 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-24T10:37:00.639 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:37:00.639 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-24T10:37:00.769 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:37:00.769 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-24T10:37:00.818 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:37:00.819 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-24T10:37:00.869 INFO:tasks.ceph:Archiving mon data... 2024-03-24T10:37:00.870 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/data/mon.a.tgz 2024-03-24T10:37:00.870 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-24T10:37:00.885 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.6rIhZ4pBmR 2024-03-24T10:37:00.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6rIhZ4pBmR 2024-03-24T10:37:01.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.6rIhZ4pBmR 2024-03-24T10:37:01.101 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.6rIhZ4pBmR is 344KB 2024-03-24T10:37:01.199 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.6rIhZ4pBmR 2024-03-24T10:37:01.214 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/data/mon.b.tgz 2024-03-24T10:37:01.214 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-24T10:37:01.231 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.tTJcmvJ1Uj 2024-03-24T10:37:01.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tTJcmvJ1Uj 2024-03-24T10:37:01.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.tTJcmvJ1Uj 2024-03-24T10:37:01.487 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.tTJcmvJ1Uj is 450KB 2024-03-24T10:37:01.607 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.tTJcmvJ1Uj 2024-03-24T10:37:01.622 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/data/mon.c.tgz 2024-03-24T10:37:01.625 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-24T10:37:01.676 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.eeS5L7fmHV 2024-03-24T10:37:01.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.eeS5L7fmHV 2024-03-24T10:37:01.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.eeS5L7fmHV 2024-03-24T10:37:01.918 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.eeS5L7fmHV is 451KB 2024-03-24T10:37:02.025 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.eeS5L7fmHV 2024-03-24T10:37:02.044 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-24T10:37:02.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T10:37:02.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T10:37:02.128 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T10:37:02.165 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T10:37:02.166 INFO:tasks.ceph:Archiving crash dumps... 2024-03-24T10:37:02.167 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi001/crash 2024-03-24T10:37:02.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T10:37:02.197 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi003/crash 2024-03-24T10:37:02.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T10:37:02.238 INFO:tasks.ceph:Shutting down logrotate 2024-03-24T10:37:02.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:37:02.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:37:02.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T10:37:02.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T10:37:02.404 INFO:tasks.ceph:Compressing logs... 2024-03-24T10:37:02.405 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:37:02.419 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:37:02.445 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59655.log 2024-03-24T10:37:02.445 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43717.log 2024-03-24T10:37:02.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.59655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49444.log 2024-03-24T10:37:02.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45625.log 2024-03-24T10:37:02.445 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49444.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43717.log.gz 2024-03-24T10:37:02.446 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51663.log 2024-03-24T10:37:02.492 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-24T10:37:02.492 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37463.log 2024-03-24T10:37:02.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43895.log 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37463.log.gz 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.1.43895.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37383.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-03-24T10:37:02.493 INFO:teuthology.orchestra.run.smithi003.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.1.43895.log.gz 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37343.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44352.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37343.log.gz 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.3.44352.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37503.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37543.log 2024-03-24T10:37:02.494 INFO:teuthology.orchestra.run.smithi003.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.3.44352.log.gz 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37503.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37503.log.gz 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37303.log 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37543.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.37543.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.37423.log 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37303.log: /var/log/ceph/ceph-mgr.x.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37303.log.gz 2024-03-24T10:37:02.495 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37423.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37423.log.gz 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mds.d.log: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-24T10:37:02.496 INFO:teuthology.orchestra.run.smithi003.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-24T10:37:02.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-24T10:37:02.615 INFO:teuthology.orchestra.run.smithi003.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-24T10:37:02.617 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49444.log.gz 2024-03-24T10:37:02.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51082.log 2024-03-24T10:37:02.617 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.0.59655.log.gz 2024-03-24T10:37:02.617 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42981.log 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45625.log.gz 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47677.log 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51663.log.gz 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51082.log.gz 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47677.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.42981.log.gz --verbose 2024-03-24T10:37:02.618 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-mon.a.log 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47677.log.gz 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57726.log 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.a.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43633.log 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.57726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59149.log 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61907.log 2024-03-24T10:37:02.619 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43633.log.gz 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54974.log 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59149.log.gz 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56321.log 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51389.log 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61907.log.gz 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54974.log.gz 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.57726.log.gz 2024-03-24T10:37:02.620 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51389.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56321.log.gz 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48055.log 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50530.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51389.log.gz 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57130.log 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54154.log 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57130.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48055.log.gz 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45201.log 2024-03-24T10:37:02.621 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57130.log.gz 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45793.log 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54154.log.gz 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61133.log 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45201.log.gz 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61133.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45793.log.gz 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56525.log 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61133.log.gz 2024-03-24T10:37:02.622 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44919.log 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56525.log.gz 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43461.log 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44919.log.gz 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59992.log 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43461.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-03-24T10:37:02.623 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51208.log 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43461.log.gz 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56695.log 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59992.log.gz 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55325.log 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56695.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51208.log.gz 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56695.log.gz 2024-03-24T10:37:02.624 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53682.log 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55325.log.gz 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54790.log 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53682.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48013.log 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53682.log.gz 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54790.log.gz 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.48013.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51705.log 2024-03-24T10:37:02.625 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57088.log 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48013.log.gz 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56567.log 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51705.log.gz 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51579.log 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56567.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57088.log.gz 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57959.log 2024-03-24T10:37:02.626 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51579.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56567.log.gz 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55703.log 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51579.log.gz 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55076.log 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57959.log.gz 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55703.log.gz 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58311.log 2024-03-24T10:37:02.627 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49344.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55076.log.gz 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47845.log 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58311.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gz 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43292.log 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58311.log.gz 2024-03-24T10:37:02.629 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61343.log 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47845.log.gz 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44581.log 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61343.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.43292.log.gz 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44581.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61343.log.gz 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49402.log 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44581.log.gz 2024-03-24T10:37:02.630 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51537.log 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49402.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56962.log 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49402.log.gz 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53824.log 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51537.log.gz 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54932.log 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53824.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56962.log.gz 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55932.log 2024-03-24T10:37:02.631 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53824.log.gz 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60684.log 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54932.log.gz 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56739.log 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60684.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.55932.log.gz 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46063.log 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56739.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60684.log.gz 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56790.log 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56739.log.gz 2024-03-24T10:37:02.632 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49218.log 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56790.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46063.log.gz 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45121.log 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56790.log.gz 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44071.log 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49218.log.gz 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59007.log 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44071.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.45121.log.gz 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59908.log 2024-03-24T10:37:02.633 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44071.log.gz 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53640.log 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59007.log.gz 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45709.log 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55577.log 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59908.log.gz 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53350.log 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45709.log.gz 2024-03-24T10:37:02.634 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48766.log 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55577.log: /var/log/ceph/ceph-client.0.53350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58043.log 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55577.log.gz 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48157.log 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48766.log.gz 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50914.log 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58043.log.gz 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59395.log 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48157.log: 91.0% -- replaced with /var/log/ceph/ceph-client.0.53350.log.gz 2024-03-24T10:37:02.635 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50914.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48157.log.gz 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55974.log 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50914.log.gz 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59091.log 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59395.log: /var/log/ceph/ceph-client.admin.55974.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.59395.log.gz /var/log/ceph/ceph-client.admin.45835.log 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55974.log.gz 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45835.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59091.log.gz 2024-03-24T10:37:02.636 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61763.log 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45835.log.gz 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45667.log 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50956.log 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45667.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61763.log.gz 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60312.log 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45667.log.gz 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52400.log 2024-03-24T10:37:02.637 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8%/var/log/ceph/ceph-client.admin.60312.log: -- replaced with /var/log/ceph/ceph-client.admin.50956.log.gz 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54622.log 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52400.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60312.log.gz 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53724.log 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54622.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52400.log.gz 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48478.log 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54622.log.gz 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51747.log 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48478.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53724.log.gz 2024-03-24T10:37:02.638 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55661.log 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48478.log.gz 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55535.log 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51747.log.gz 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55535.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55661.log.gz 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58353.log 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55535.log.gz 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-03-24T10:37:02.639 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57046.log 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50362.log 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58353.log.gz 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57046.log.gz 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2024-03-24T10:37:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54112.log 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50362.log.gz 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60270.log 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47593.log 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54112.log.gz 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60270.log.gz 2024-03-24T10:37:02.641 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45961.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47593.log.gz 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59829.log 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51855.log 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59829.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44579.log 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48604.log 2024-03-24T10:37:02.642 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51855.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59829.log.gz 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48646.log 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51855.log.gz 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48604.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44579.log.gz 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58185.log 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48604.log.gz 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44247.log 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48646.log.gz 2024-03-24T10:37:02.643 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53766.log 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44247.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58185.log.gz 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44247.log.gz 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46607.log 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53766.log.gz 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61553.log 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46607.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49528.log 2024-03-24T10:37:02.644 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61553.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46607.log.gz 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49570.log 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61553.log.gz 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49528.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49570.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49528.log.gz 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48097.log 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49570.log.gz 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47719.log 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43335.log 2024-03-24T10:37:02.645 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47719.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48097.log.gz 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54890.log 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47719.log.gz 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52618.log 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54748.log 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43335.log.gz 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52618.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54890.log.gz 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58965.log 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T10:37:02.646 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54706.log 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54748.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52618.log.gz 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53908.log 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54748.log.gz 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54706.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58965.log.gz 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44569.log 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54706.log.gz 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-03-24T10:37:02.647 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44569.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53908.log.gz 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56016.log 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44569.log.gz 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55367.log 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44248.log 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55367.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.56016.log.gz --verbose 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.61091.log 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55367.log.gz 2024-03-24T10:37:02.648 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55118.log 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44248.log.gz 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51296.log 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61091.log.gz 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51296.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.55118.log.gz 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2%/var/log/ceph/ceph-client.admin.43819.log: -- replaced with /var/log/ceph/ceph-client.admin.51296.log.gz 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53598.log 2024-03-24T10:37:02.649 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61301.log 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60228.log 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61301.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.53598.log.gz 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56609.log 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60228.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61301.log.gz 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53950.log 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60228.log.gz 2024-03-24T10:37:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60624.log 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49984.log 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56609.log.gz 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60624.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53950.log.gz 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57358.log 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60624.log.gz 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57358.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49984.log.gzgzip 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58923.log 2024-03-24T10:37:02.651 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57358.log.gz 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52358.log 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61049.log 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58923.log.gz 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44251.log 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52358.log.gz 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44253.log 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61049.log.gz 2024-03-24T10:37:02.652 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44253.log: gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44251.log.gz 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.53556.log 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43777.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44253.log.gz 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56184.log 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56483.log 2024-03-24T10:37:02.653 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53556.log.gz 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56184.log.gz 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58127.log 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56874.log 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58127.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56483.log.gz 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54034.log 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58127.log.gz 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61637.log 2024-03-24T10:37:02.654 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54034.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45457.log 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61637.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54034.log.gz 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60186.log 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61637.log.gz 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55890.log 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61217.log 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60186.log.gzgzip 2024-03-24T10:37:02.655 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48850.log 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55890.log.gz/var/log/ceph/ceph-client.admin.61217.log: 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60034.log 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61217.log.gz 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57274.log 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60034.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48850.log.gz 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57274.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60034.log.gz 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2024-03-24T10:37:02.656 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57274.log.gz 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45919.log 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49018.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz -5 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60853.log 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45919.log.gz 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60522.log 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44245.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60853.log.gz 2024-03-24T10:37:02.657 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48294.log 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60522.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31826.log 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60522.log.gz 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48294.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.31826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53866.log 88.1% 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48294.log.gz 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31826.log.gz 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44588.log 2024-03-24T10:37:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59049.log 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53866.log.gz 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46484.log 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44588.log.gz 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48520.log 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59049.log: /var/log/ceph/ceph-client.0.46484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44580.log 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59049.log.gz 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51495.log 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48520.log.gz 2024-03-24T10:37:02.659 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48934.log 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.46484.log.gz 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60438.log 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51495.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44580.log.gz 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56226.log 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51495.log.gz 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60438.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48934.log.gz 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52534.log 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60438.log.gz 2024-03-24T10:37:02.660 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51166.log 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56226.log: /var/log/ceph/ceph-client.admin.52534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56226.log.gz 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51166.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52534.log.gz 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55787.log 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51166.log.gz 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61595.log 2024-03-24T10:37:02.661 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49648.log: /var/log/ceph/ceph-client.admin.55787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50278.log 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55787.log.gz 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57466.log 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50278.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61595.log.gz 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51347.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.50278.log.gz 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43547.log 2024-03-24T10:37:02.662 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57172.log 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57466.log: /var/log/ceph/ceph-client.admin.43547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57466.log.gz 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57172.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43547.log.gz 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61947.log 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45751.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57172.log.gz 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58865.log 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2024-03-24T10:37:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44785.log 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61947.log: /var/log/ceph/ceph-client.admin.58865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47803.log 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61947.log.gz 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44785.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58865.log.gz 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44785.log.gz 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47803.log: /var/log/ceph/ceph-client.admin.47467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61259.log 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47803.log.gz 2024-03-24T10:37:02.664 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52476.log 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58269.log 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61259.log: /var/log/ceph/ceph-client.admin.52476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55619.log 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61259.log.gz 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58269.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52476.log.gz 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53012.log 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56142.log 2024-03-24T10:37:02.665 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58269.log.gz 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53012.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55619.log.gz 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43903.log 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53012.log.gz 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56142.log: /var/log/ceph/ceph-client.admin.43903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50706.log 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56142.log.gz 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45141.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47425.log 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43903.log.gz 2024-03-24T10:37:02.666 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50573.log 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50706.log.gz 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50625.log 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47425.log: /var/log/ceph/ceph-client.admin.50573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47929.log 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47425.log.gz 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50625.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50573.log.gz 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52928.log 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50625.log.gz 2024-03-24T10:37:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56100.log 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47929.log: /var/log/ceph/ceph-client.admin.52928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43169.log 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47929.log.gz 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52928.log.gz 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58431.log 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56100.log.gz 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47257.log 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43169.log: /var/log/ceph/ceph-client.admin.58431.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43169.log.gz 2024-03-24T10:37:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47257.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58431.log.gz 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55034.log 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56832.log 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.3.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47257.log.gz 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52576.log 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55034.log.gz 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43861.log 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57004.log 2024-03-24T10:37:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52576.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56832.log.gz 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60769.log 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52576.log.gz 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57004.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43861.log.gz 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51040.log 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50236.log 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57004.log.gz 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51040.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60769.log.gz 2024-03-24T10:37:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50236.log: gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.admin.51252.log 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51040.log.gz 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44243.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50236.log.gz 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54414.log 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48724.log 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.54414.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51252.log.gz 2024-03-24T10:37:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61427.log 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48724.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gzgzip 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38270.log 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61427.log: gzip -5 --verbose 87.4% -- /var/log/ceph/ceph-client.admin.60811.log -- replaced with /var/log/ceph/ceph-client.admin.48724.log.gz 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61427.log.gz 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr: --/var/log/ceph/ceph-client.admin.38270.log: /var/log/ceph/ceph-client.admin.45218.log 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60811.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38270.log.gz 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60811.log.gz 2024-03-24T10:37:02.672 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose 91.3% -- replaced with /var/log/ceph/ceph-client.0.54414.log.gz 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.61385.log 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48808.log 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45218.log.gz 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52115.log 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61385.log.gz 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59191.log 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.0.52115.log: -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48808.log.gz 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44924.log 2024-03-24T10:37:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59191.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59191.log.gz 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43675.log 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49816.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44924.log.gz 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.44250.log 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48892.log 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44250.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.52115.log.gz 2024-03-24T10:37:02.674 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45499.log 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43675.log.gz 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44250.log.gz 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45877.log 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61679.log 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45499.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48892.log.gz 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51789.log 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45499.log.gz 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45877.log.gz 2024-03-24T10:37:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57214.log 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59233.log 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51789.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61679.log.gz 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57316.log 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59233.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57214.log.gz 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60480.log 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.51789.log.gz 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57316.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59233.log.gz 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45415.log 2024-03-24T10:37:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60480.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57316.log.gz 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52886.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60480.log.gz 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49302.log 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3%/var/log/ceph/ceph-client.admin.52886.log: -- replaced with /var/log/ceph/ceph-client.admin.45415.log.gz 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50320.log 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49302.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61469.log 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52886.log.gz 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49302.log.gz 2024-03-24T10:37:02.677 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58001.log 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50998.log 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61469.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44922.log 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61469.log.gz 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6%/var/log/ceph/ceph-client.admin.50998.log: -- replaced with /var/log/ceph/ceph-client.admin.58001.log.gz 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47971.log 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44922.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50998.log.gz 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59275.log 2024-03-24T10:37:02.678 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47971.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44922.log.gz 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47971.log.gz 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47147.log 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59275.log.gz 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43419.log 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47383.log 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43419.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.47147.log.gz 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48336.log 2024-03-24T10:37:02.679 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52316.log 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48378.log 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47383.log.gz 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50194.log 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48336.log.gz 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52316.log: /var/log/ceph/ceph-client.admin.48378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56058.log 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52744.log 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48378.log.gz 2024-03-24T10:37:02.680 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52316.log.gz 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48252.log 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50194.log.gz 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56058.log.gz 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55409.log 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52744.log: /var/log/ceph/ceph-client.admin.48252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49858.log 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52744.log.gz 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55451.log 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48252.log.gz 2024-03-24T10:37:02.681 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55409.log.gz 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49858.log: --verbose -- /var/log/ceph/ceph-client.admin.49900.log 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55451.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49858.log.gz 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47105.log 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55451.log.gz 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45583.log 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49900.log.gz 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52702.log 2024-03-24T10:37:02.682 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45583.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47105.log.gz 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43250.log 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45583.log.gz 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53090.log 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52702.log.gz 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58085.log 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43250.log.gz 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44923.log 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53090.log.gz 2024-03-24T10:37:02.683 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58085.log.gz 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56651.log 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44923.log.gz 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57400.log 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59317.log 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57400.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56651.log.gz 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43377.log 2024-03-24T10:37:02.684 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59317.log: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57400.log.gz 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59317.log.gz 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49486.log 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43377.log.gz 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58691.log 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47635.log 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49486.log: /var/log/ceph/ceph-client.0.58691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47551.log 2024-03-24T10:37:02.685 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47635.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49486.log.gz 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47635.log.gz 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50828.log 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47551.log: 89.8% -- replaced with /var/log/ceph/ceph-client.0.58691.log.gz 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47551.log.gz 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43591.log 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50828.log: /var/log/ceph/ceph-client.admin.48562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47761.log 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43591.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50828.log.gz 2024-03-24T10:37:02.686 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43591.log.gz 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61989.log 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47761.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50786.log 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61989.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47761.log.gz 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46983.log 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T10:37:02.687 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50786.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50786.log.gz 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43503.log 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.2.46983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51621.log 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43503.log: 87.5%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.43503.log.gz 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44252.log 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.2.46983.log.gz 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61847.log 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44252.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51621.log.gz 2024-03-24T10:37:02.688 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44252.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60727.log 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60727.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61847.log.gz 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60727.log.gz 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50068.log 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50870.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50068.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50068.log.gz 2024-03-24T10:37:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61175.log 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47341.log 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61175.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61175.log.gz 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50152.log 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47341.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47341.log.gz 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48420.log 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50152.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50152.log.gz 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54664.log 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48420.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48420.log.gz 2024-03-24T10:37:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38177.log 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54664.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54664.log.gz 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52802.log 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38177.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38177.log.gz 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37901.log 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52802.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52802.log.gz 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52844.log 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.37901.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37901.log.gz 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61805.log 2024-03-24T10:37:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52844.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47887.log 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61805.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61805.log.gz 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45208.log 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47887.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47887.log.gz 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44589.log 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45208.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45208.log.gz 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55745.log 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44589.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2024-03-24T10:37:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49942.log 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55745.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55745.log.gz 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60076.log 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49942.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49942.log.gz 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60076.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60076.log.gz 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61721.log 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51435.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51124.log 2024-03-24T10:37:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61721.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61721.log.gz 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58227.log 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51124.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51124.log.gz 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58227.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58227.log.gz 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55196.log 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45541.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56421.log 2024-03-24T10:37:02.694 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55196.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55196.log.gz 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52660.log 90.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56421.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56421.log.gz 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56920.log 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52660.log.gz 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57901.log 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56920.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-03-24T10:37:02.695 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61007.log 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56920.log.gz 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49690.log 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.61007.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57901.log.gz 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61007.log.gz 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.49690.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55847.log 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52272.log 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49690.log.gz 2024-03-24T10:37:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55847.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55847.log.gz 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55493.log 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47299.log 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52272.log: /var/log/ceph/ceph-client.admin.55493.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52272.log.gz 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55493.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60354.log 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.47299.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47299.log.gz 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61511.log 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52970.log 2024-03-24T10:37:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60354.log: /var/log/ceph/ceph-client.admin.61511.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60354.log.gz 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 88.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.43073.log 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61511.log.gz 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52970.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52970.log.gz 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60564.log 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59950.log 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43073.log: /var/log/ceph/ceph-client.admin.60564.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43073.log.gz 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60564.log.gzgzip 2024-03-24T10:37:02.698 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53992.log 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59950.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59950.log.gz 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44918.log 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53992.log: /var/log/ceph/ceph-client.admin.44918.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53992.log.gz 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44918.log.gz 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.60396.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2024-03-24T10:37:02.699 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-24T10:37:02.700 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-24T10:37:02.700 INFO:teuthology.orchestra.run.smithi001.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-24T10:37:02.701 INFO:teuthology.orchestra.run.smithi003.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-24T10:37:02.701 INFO:teuthology.orchestra.run.smithi003.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.226s 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m1.045s 2024-03-24T10:37:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.061s 2024-03-24T10:37:02.722 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-24T10:37:02.755 INFO:teuthology.orchestra.run.smithi001.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-24T10:37:02.773 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-24T10:37:02.774 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-24T10:37:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m0.343s 2024-03-24T10:37:02.774 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m1.326s 2024-03-24T10:37:02.775 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.231s 2024-03-24T10:37:02.775 INFO:tasks.ceph:Archiving logs... 2024-03-24T10:37:02.775 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi001/log 2024-03-24T10:37:02.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T10:37:03.289 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi003/log 2024-03-24T10:37:03.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T10:37:03.455 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T10:37:03.480 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T10:37:03.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:37:03.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:37:03.560 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T10:37:03.576 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T10:37:03.577 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:37:03.579 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:37:03.591 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:37:03.594 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +259us[ +259us] +/- 73ms 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -468us[ -448us] +/- 53ms 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +396us[ +396us] +/- 75ms 2024-03-24T10:37:03.595 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:37:03.599 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:37:03.599 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-24T10:37:03.599 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +259us[ +259us] +/- 73ms 2024-03-24T10:37:03.599 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -470us[ -457us] +/- 53ms 2024-03-24T10:37:03.600 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +382us[ +382us] +/- 75ms 2024-03-24T10:37:03.600 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:37:03.601 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T10:37:03.610 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T10:37:03.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T10:37:03.621 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:37:03.715 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:37:03.785 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:37:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:37:03.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:37:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:37:03.845 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T10:37:03.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T10:37:03.875 INFO:teuthology.task.internal:Duration was 646.493063 seconds 2024-03-24T10:37:03.875 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T10:37:03.885 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T10:37:03.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:37:03.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:37:03.928 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:37:03.931 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:37:04.103 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T10:37:04.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-24T10:37:04.104 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T10:37:04.163 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-24T10:37:04.163 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T10:37:04.189 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T10:37:04.190 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T10:37:04.205 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T10:37:04.258 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T10:37:04.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T10:37:04.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T10:37:04.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T10:37:04.409 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T10:37:04.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:37:04.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:37:04.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T10:37:04.473 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:37:04.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:37:04.501 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-24T10:37:04.527 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-24T10:37:04.544 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:37:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:37:04.573 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:37:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:37:04.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T10:37:04.609 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T10:37:04.610 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi001 2024-03-24T10:37:04.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:37:04.652 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049/remote/smithi003 2024-03-24T10:37:04.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:37:04.688 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T10:37:04.688 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T10:37:04.691 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T10:37:04.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T10:37:04.749 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T10:37:04.749 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T10:37:04.760 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T10:37:04.761 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:37:04.764 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:37:04.776 INFO:teuthology.orchestra.run.smithi001.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:37 /home/ubuntu/cephtest 2024-03-24T10:37:04.791 INFO:teuthology.orchestra.run.smithi003.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:37 /home/ubuntu/cephtest 2024-03-24T10:37:04.792 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T10:37:04.804 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T10:37:04.825 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T10:37:04.852 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-24T10:37:04.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:37:04.876 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-24T10:37:04.876 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:37:04.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:17:57.552395', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI9EvjocjrvBu7amPMhnD4sgSivbF05PbumcxFiTQTEAOnMr9s+stLL13DGeLbt4DBy4k42+xo17vHsmObc4vbo='} 2024-03-24T10:37:04.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:17:57.554437', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCdFtkHL/LzKO/bqO1/8DiX80bAYuDCkHNfY0y+tyyywsuYpE5ok+2ZrGqubxi+op9W4OeJOGeX//+Qdh0erhwU='} 2024-03-24T10:37:04.942 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-24T10:37:04.942 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:37:04.971 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-24T10:37:04.972 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:04.989 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-24T10:37:04.990 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:37:05.003 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-24T10:37:05.018 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-24T10:37:05.018 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:05.043 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-24T10:37:09.005 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:09.045 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:09.117 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-24T10:37:09.156 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-24T10:37:13.118 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:13.158 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:37:13.230 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-24T10:37:13.274 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-24T10:37:13.333 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-24T10:37:13.375 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-24T10:37:13.480 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} duration: 646.4930634498596 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bbf6487d0e7c4f76a436a5205f5b71c9 status: fail success: false 2024-03-24T10:37:13.480 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:37:13.577 INFO:teuthology.run:FAIL